Tracking monitoring

Introduction

Monitoring the tracking projects which Catinet are running consists of several tasks which must be performed during the tracking period:

• Daily management and tracking overview reports
  
• Fixing quotas / changing quota periods
• Quota standards for the Automated Tracking Script
• Setting up tracking management of new tracking questionnaires
• Group standards
• Setting up a tracking for external sample supply
• Managing a tracking with external sample supply
• Setting up tracking management of new tracking questionnaires
• Group standards (AGAIN?!)
• Selecting sample
  

Each task is described in details in the following sections.

Daily tracking management

All trackings must be under supervision Monday – Friday. Usually every tracking must be open and complete interviews as evenly as possible every one of these week days, before the weekly target is reached. Furthermore, it is of the utmost importance that the weekly target is reached Sunday at the absolute latest. Reaching the weekly target Friday is preferred.

To a tracking manager, a tracking consists of the following relevant items

• a questionnaire
  • with quotas
  • with a bulkmail
• a set of groups, usually the following:
  • a mailout group
  • a samplestatus group
  • an exclude group
• a work flow (an “Automated Tracking Script”)
• a scheduled cattask

This is at the moment all supervised with a master document, the “Tracking Overview”, which is used for supervising and creating tracking reports.

Regardless of who is handling the trackings (in Denmark or in Vietnam), the workflows’ scheduled cattasks should always be set to run between 10:00 and 11:00 Danish time

There are usually some changes to be made to some of the tracking questionnaires, and sometimes we do not know before Monday morning at 09:00-09:45. In that case, we need time to be able to set the respective tracking questionnaires on “paused” status, so that the respective scheduled cattask will not execute sample.

When the last tracking cattask has run to end, the daily tracking managent can begin. This would be 10 minutes after the time of the last scheduled tracking cattask.

The “tracking overview” is copied into a “tracking overview reports” folder, and is renamed to “week XX weekday tracking overview”(for instance “week 51 Tuesday tracking overview”). This folder will then hold historical data of the progress and handling of all trackings, which can be used for error correction, optimization, and gaining experience and knowledge. I have gained a lot by this method. Also it allows others to pick up where the work was left.

Open the copied tracking overview report for today.
Check your e-mail inbox, preferably you have made a folder specifically for tracking status reports automatically sent by the automated tracking scripts (work flows). Here you should see an unread mail for each tracking, containing information about how many mail invitations has been sent out, what targets are for the week, and what is the current status of the tracking interviews.

The three things needed to be done are:

1. Check that the target in todays tracking overview report is the same as in the status report mails for each individual tracking
2. Type in the “Current” value for each tracking in the tracking overview reports “Current” column. You should then have an overview of how many interviews are missing for each tracking in numbers along with the completion percentage of the tracking in the “progress” column to the right.
3. Double-check that all
   

Optimization:
It would be an advantage to create an automated mail that contains the “current” status of all the trackings week quota at minimum, to be sent to the tracking management group at least twice a day. This will replace having to open each individual tracking questionnaire to follow the progress.

Mondays: Tracking adjustments and corrections

Monday before 9:30 Danish time, it is important to check with the Production Manager in Denmark whether or not there are changes scheduled to be made to the tracking questionnaires for the current week. If there are changes to be made, ensure that the trackings for maintenance were indeed fully completed last week.
Always check the last status report from Sunday, to check if all trackings that were not completed Friday have been completed by Sunday. If some trackings were not completed Sunday by the time of the status report, manually open each of these questionnaires to check if last weeks quotas were met.

If they were, set the respective tracking questionnaires status on “paused”, so that the respective scheduled cattask will not execute sample before the changes to the tracking questionnaire have been made.
If they were not, this is bad. Contact the Danish Production Manager (Produktionschef), for a solution. Normally the missing interviews must be completed early Monday, before the changes to the tracking questionnaire can be made. This is a bad solution, since it greatly affects quality and procedure, as well as panellists, who cannot enter the survey shortly after being invited to it, due to the fact that the questionnaire cannot be open whilst potential changes are being made. Furthermore, it takes up an amount of data engineer time, and it is methodically wrong.

When changes have been made, tracking management will receive a “go” for the respective trackings from the Production Manager.
- Then change the tracking questionnaire status to “open” via the questionaire list (Tools  Surveys  Questionnaires)
- go to “work flow” (Tools  Administration  Work flow)
- open the respective tracking work flow, and go to the “Script” tab
- Click “Run” (do not click Run batch). The Automated Tracking Script will then send out first batch, instead of waiting untill tomorrow for the scheduled cattask to execute, thus improving tracking data quality.

Fixing quotas/changing quota periods

Quotas in the tracking questionnaire are used by the Automated Tracking Script to adjust the amount of mail invites sent to each segment of a target group of panellists, in accordance to the current amount of successful interviews. Quotas define the final amount of intervews succeeded in each individual segment of the target group, within the given quota period.
At times, the quotas must be changed or adjusted, either by wish of the Client, or for maintenance purposes. The quotas should never be changed without prior consent of the project leader.

In order for the automated tracking script to successfully address the specified quotas, quota label naming must be consistant.
- The week quotas must be labelled “YYYYWW” (example: “200913” for week 13 in the year 2009)
All remaining subquota labels must consist of a definition of time for the quotas to be active, followed by the quota name itself, like so:
- "YYYY WW - YYYY WW: segment name"
(even when the quota period is one week, in example "2009 21 - 2009 21: Males 15-25")
furthermore recommend always using segment name standards:
- “Sex AA-AA” in example: “Males 25-34” or “Females 55-99”
or in the rarer occasions where three suffixes are used in the Automated tracking Script to more precisely control the sample selection:
- “Sex AA-AA region xx” example “Males 15-25 region Hovedstaden”
Notice that these standards for quota names (not the entire quota labels) counts for mailout group rule labels just as well.

Setting up a tracking for external sample supply

All questionnaires and trackings created from the CGtemplates already have the relevant questions and scripts for redirect implemented. So when we need external sample supply, there are only three things that needs to be done:

1) A sample cost & feasibility check request must be sent to at least one Panel Partner. A supplier is then chosen, from cost, quality and feasibility etc.
Research Now and Toluna have been favourites in the last half of the year 2008, Research Now especially in the nordic countries. Which Panel Partner is best at what kind of request in quality and price, takes an amount of experience to know. There are too many factors to take into consideration to give the proper insight here.

Help: The document "International Panel Overview.xls" is useful for quickly finding out which panel partners have panels in certain countries, and of what sizes the respective panels are (These numbers are practically worth nothing, without proper experience and inside knowledge of the respective companies, most are directly far from true).
The document "Sample Request Form.xls" is used for remembering filling out all necessary information to the Panel Partners when requesting samle cost and feasibility.

2) Receive and insert exit links from the Panel Partner.
Exit links ensure that the Panel Partners panellists are returned to their own site upon completion (succession, screenout or quotafull) of the questionnaire, thus giving the Panel Partner the option of following the progress of completion of sample, or just registering the counts for all three end statuses. This depends on their own respective research platforms.

How to: The questions in the questionnaires made from the CGTemplate (questionnaire templates) needed to insert exit links into are named "SUCCEEDED_PP", "QUOTAFULL_PP" and "SCREENOUT_PP". Insert the exit links in the respective question scripts under the Questionnaire Editor --> Properties --> Edit question properties. Remember to insert the right exit links under the right end questions, and to insert the exit links in the script for the current Panel Partner! You insert the exit link (exit URL) by replacing the text INSERTXXXXXXXXXXRETURNURL inside the quotation marks with the given exit URL (still leaving the quotaion marks in the script.

3) Generate a Test link and a Live link with the correct link parameters, and send it to the Panel Partner.

*Remember that it is good custom to test the Test link yourself, appending a user id you either get from the Panel Partner, or that you know have a structure that their system will accept, and remember to inform them of the user id's you tested with. Easy examples could be "66666666" or "11111111". Assuming that the questionnaire itself has already previously been tested, you must focus on testing if the return links work, and do indeed redirect you to the Panel Partners end screens, for both Quotafull (if possible), screenout, and successfull (generally known as "completed").

Ensure that the Panel Partner does not use the Live link for testing, as live tests take time to locate in Catglobe, and convert to test interviews. Test links are exactly the same as live links, with the two exception that a test link can be used multiple times, and has a "test" link parameter, that automatically converts what is in fact really a live interview to a test interview at the end of the survey.

How to
• in the questionnaire list, locate and highlight the relevant questionnaire
• click "Actions" --> "Information"
• copy the entire "Anonymous survey hyperlink" (the light green url)
• paste it into the worksheet of the current project in the "Tracking Overview.xls" master document (so that this information remains in all following Tracking Overview Reports
• Now, to get the Test link edit the anonymous survey hyperlink like so (always assume case sensitivity):
o change the link parameter "&n=1" to "&n=2" (always without the quotation marks)
o insert the Bypasscookie parameter at the end, like this "&b=1" (this bypasses a cookie, and lets you use the link for testing multiple times).
o Insert the Test parameter "&Test=1" (converts the otherwise live interview to a test interview instead)
o insert the Panel_Provider parameter "&PP=?". To know exactly what to write instead of the question mark, open the questionnaire with the Questionnaire Editor, and go to the question "Panel_Provider". Here you see a list of panel partner names. Find the relevant panel partner and type in the name exactly as it is written. The parameter could for instance be "&PP=Toluna"
o insert the Panel_Provider_User_ID. This is a specific link parameter for each panel partner. To know exactly what to write, open the questionnaire with the Questionnaire Editor, and go to the question "Panel_Provider_User_ID". Find the relevant panel partner user id parameter and type it in exactly as it is written. The parameter could for instance be "&TolunaID="
• Now you have a Test link that could for instance look like this: "http://mycatinet.catglobe.com/Login.aspx?r=93c1b14e-f989-4c26-b180-82fba92c375b&n=2&b=1&Test=1&PP=Toluna&TolunaID=". The Panel Partner must insert an existing ID to be sure to be returned to their exit links when testing.
• Now, to get the Live link edit the Test link like so:
o remove the Test parameter (so the interview remains a live interview)
o remove the Bypasscookie parameter (so the link can be used only once)
• Now you have a corresponding Live link that would look something like this: "http://mycatinet.catglobe.com/Login.aspx?r=93c1b14e-f989-4c26-b180-82fba92c375b&n=2&PP=Toluna&TolunaID=". The Panel Partner must be instructed to append their panellist user id at the end of the live link, right after the "=" mark (without quotation marks)

If just one parameter is wrong, it is likely to have bad consequenses. The interviews might seemingly succeed with valid data, but really be test interviews, or we might lose all id's from the Panel Partner. This would mean they are unable to incentivise, and that we have no proof of the amount of completes they have delivered.

More from Sebastian on Wed, Jan 14, 2009 (Hòa will edit this section)

If a trackings quotas were not met at the end of the week: If tracking quotas were not met Friday last week or during the weekend, it is important to find out why. Firstly, when having typed in the current-values of Sunday's Automated Mailout reports from Catglobe, you must open the questionnaire which quotas seemingly were not met by Sunday, and check the week quota from last week. More interviews should have been completed since during Sunday after the automated mailout. If the questionnaires week quota still was not met, you must find the reasons for this.

It could be one or more of the following reasons: - We have spent all samples in one or more of the target groups (the youngest male target group is definitely the first place to look, due to the content of our internet panels) - Catglobe had a bad moment and did not send out - There is one ore more errors in the quotas (in the expressions, in the targets, or in the general existence of quotas) - a large number of other less likely scenarios

Do the following: Check to see the spread of the missing interviews in the sub quotas. If there is a cluster of missing completes in one of the sub quotas, there might be a shortage of sample in that target group. It would most likely be males, and it would most likely be the youngest target group. Depending on the general target group of the survey, that might not be the case, but please apply logic in your search, bearing in mind that Catinets Internet Panel has fewer males, and fewer young (15-25 years) and old (60+ years) panellists.

To check if it is true that there is a shortage of sample, go to Catglobes Groups page, and search for "samplestatus,mailout group,exclude group" in group name, with the search criteria "start of name". You then have all groups that should be related to tracking projects, if all group naming standards were followed.

Find the samplestatus group assigned to the relevant tracking project, open it, and rebuild it (if it is not scheduled for regular rebuild). When it has been rebuilt, check what sample there is left. If there is 0-10 in one of the target groups, it certainly is a shortage of sample, and this must be reported to the Production Manager, and a solution must be found by either finding external sample supply through our partners, or working out an alternative with the client.

Meanwhile, to get the project going for as long as possible, open the mailout group assigned to the relevant tracking project, and open the rule of the target group that has run out of sample. Now, to retrieve more sample, we change this rule into being a reminder rule, meaning that respondents that has already received an invitation for the tracking, but has not started or finished the survey will be invited again, and hopefully complete the survey. This is done by editing the Usage Variable for the relevant questionnaire. Before editing, only the box "Users who did not participate in the following questionnaire" should be ticked/marked. Change that by removing that x, and ticking/marking these two: "Users who partly completed the following questionnaire" and "Users who did not start the following questionnaire". Before saving and closing, please rename the mailout group, so that all can see it is part reminders. (add " - temporarily edited for reminders" or such). This will only help one week or so, if even that much. Therefore, it is important to regularly check all samplestatus groups! This will ensure that you are ahead of any issues on the way.

If the spread appears to be even, take a good, long, hard look at the quotas, and apply logic. Does one quotas target limit the other? If 98 women are needed and 102 men are needed in a one week quota period, but the week quota is 150, something is obviously wrong with the quotas. Consult with the production manager regarding these issues. Not all sub quotas are supposed to be reached 100%. Check the tolerance level of the sub quotas in the quota description column. Gender is generally zero tolerance level, Age is often 10% tolerance level, and region is often 10% tolerance as well. Week quotas are always zero tolerance. If there is a 10% tolerance level in the age quotas, the sum of the age sub quotas will be 110% of the week quota, per week in the specified quota period. The interviews may fall into the age sub quotas as naturally as they will. three age quota can be met 100%, and another only 70%.

Compensating for last weeks missing completes

The unofficial partial solution to fixing a tracking which quotas were not met: Gathering interviews in the week after the one week where the quotas were not met, and later moving the respondents to the previous week. This is methodically wrong, but has proven to be a method preferred to buying external sample at Catinét, and most likely other market research companies. Keep in mind, that it is important the extra completes gathered match the profiles of the missing completes.

Example of a week with missing completes:

Quotas: Target Current Description

200907 100 87 week quota 200907 - 200907: Males 46 35 zero tolerance - gender quota 200907 - 200907: Females 54 52 zero tolerance - gender quota 200907 - 200907: 15-25 27 7 10% tolerance - age quota 200907 - 200907: 26-35 43 39 10% tolerance - age quota 200907 - 200907: 36-50 41 41 10% tolerance - age quota 200907 - 200907: DK East 61 44 10% tolerance - regional quota 200907 - 200907: DK West 50 43 10% tolerance - regional quota

Example of adjustments to the following week with compensation

Quotas: Target Current Description

200908 113 3 week quota, +13 for w07 200908 - 200908: Males 57 0 zero tolerance - gender quota, +11 for w07 200908 - 200908: Females 56 3 zero tolerance - gender quota, +2 for w07 200908 - 200908: 15-25 40 0 10% tolerance - age quota, +13 for w07 200908 - 200908: 26-35 43 2 10% tolerance - age quota 200908 - 200908: 36-50 41 1 10% tolerance - age quota 200908 - 200908: DK East 71 3 10% tolerance - regional quota, +10 for w07 200908 - 200908: DK West 53 0 10% tolerance - regional quota, +3 for w07

The quotas that need adjustments are opened and edited. Make sure that the description is edited too. Always add a description of the increase in numbers, and the week the extra interviews are to be moved to. Later on (it might be 2-4 weeks) a data consultant in Denmark will by script manipulate the completion timing of the interview according to the description of the quotas. If there is no description, this takes up a long time for the data consultant.

This only helps what has already happened. Changes need to be made to ensure that the tracking project goes according to plan.