Configuring Channels and Datafeeds

In IntelliPortal, a channel is a path to an end-user client machine and a datafeed is a path to cache or to an external data source. An adapter is an interface between the datafeed and an extension that actually retrieves the external data. This section explains how to create channel and datafeed records for channels that are built by developers.

Adapters and datafeeds work together and can be viewed as a single entity. (This is explained in more detail in Adapters, Datafeeds, and Channels.) The Server Administrator is the application used to manage these channels and datafeeds. It uses channel records and datafeed records to manage these objects. For this to happen, you need to create corresponding channel records and datafeed records using the Server Administrator.

• You must create a channel record for any channel that has a datafeed, that is for any channel that accesses external data. If a channel doesn't have a datafeed, the channel record is optional.
• You must create a datafeed record for every datafeed.

Note that all channels can use datafeeds; they are not confined to Java channels. For example, an HTML channel can be configured to fetch new HTML pages from an external source at regular intervals. It needs a datafeed to do this.

The Configurators menu item on the Server Administrator lets you dynamically add or change channel and datafeed records and commit them to a server. A key feature of IntelliPortal is that you can dynamically add or change channels and datafeeds without starting or stopping the server.

___________________________________

Topics in this section

• Configuration Overview
• Creating Channel Records
• Creating Datafeed Records
• Configuration Examples

Configuration Overview

All of the following basic steps are typically needed to configure and publish a channel that can access external data. Many of the steps may be done concurrently, by different developers, and the actual sequence may vary, but each step is required.

1. Define your requirements in the end user channel to which a user may subscribe.
2. Decide what the client and adapter exchange (Strings, serialized objects, arrays of objects, etc.).
3. Write the adapter code (if a pre-built adapter is not adequate) as explained in the Channel Developer Guide.
4. Add the adapter to the outside_classes directory on the server and then associate it with a datafeed using the Server Administrator.
5. Again using the Server Administrator, add a channel record for the channel and a datafeed record for the channel.
6. Write the Java code for the channel.
7. Publish and test the channel.

How Adapters Work

To create the adapter, you implement IAdapter as explained in the Channel Developer Guide; this makes the adapter usable by the datafeed. The backend of the adapter (the extension) is custom code, created by a channel developer. Then you run the Server Administrator and enter specific metadata such as the datafeed name, how often the data changes, etc.

All external adapter files must reside in the outside_classes directory which is created on the server when the system is installed. This directory includes pre-built adapters URLAdapter and FileReaderAdapter and is also used for additional adapters created by developers. The Java code that implements IAdapter must be written and added to this directory; this is where the server looks for external adapters.

There are three interrelated steps in adapter development:

1. Write a new adapter (or use a pre-built adapter) for access to external data.
2. Put the adapter file in the outside_classes directory.
3. Configure the channel and datafeed records.

When the IntelliPortal server starts, it first reads the property files in the user's etc directory. These files provide information on such things as how to connect to the database, proxy information, and various startup parameters. Next, it reads the metadata (if applicable). Metadata is data about data. For transmitters, this includes information about channels, adapters, and the characteristics of data being stored on the server. This metadata is configurable and can be dynamically changed while the server is running. This means you can add, delete, and modify channels and datafeeds, and their behavior, without stopping the server.

When you commit a change, the metadata is changed, any interested parties (i.e. running transmitters) are notified, and the changes are logged. This log is located in the IntelliPortal logs directory and named MetaDataChanges.log. It is written in SQL format so that it can be used to re-apply these same administrative changes at some later time.

Data That Adapters Retrieve

During channel updates, various types of data are passed from the server to the client. When configuring channel records and datafeed records as explained on the following pages, it is important to understand the different types of data that are accessible from the server during channel updates.

Creating Channel Records

Channel configuration is closely associated with the adapter and datafeed configuration. Adapters are used by channels to fetch data from outside the IntelliPortal server; datafeeds manage that data once it has been fetched. This section explains channel records and their association with datafeeds.

A channel can access data through datafeeds to provide Channel Custom Data and User Specific Data to a client (and Usage Tracking Data if desired). Using this metadata, any number of datafeeds can be associated with a channel. They can be grouped together for convenience, and different chunks of data can be sent to different datafeeds in a defined group. Without programming and using a channel's associated datafeed, a client may send the following to adapters:

• Strings
• Strings parsed into tokens based on a delimiter determined by the client
• Strings and tokenized strings to subsets (or groups) of datafeeds (and adapters)
• Objects
• Different objects to different subsets of datafeeds

Additionally, once the data is fetched, it is placed in a file whose name is determined by the metadata. This name can be automatically modified and made unique for multiple, indeterminate numbers of returned files. Any exceptions on the server that can be caught are placed in UpdateExceptions.txt and returned for debugging.

As show in the Channel Configurator window, channel records have two parts.

• Static Channel Data - There is only one set of channel data per channel. This data applies to the channel as a whole.
• Dynamic Channel Data - There are zero to any number of these records per channel, and this data represents a datafeed's (and its adapter's) association with the channel. This is the mechanism for sending Channel Custom Data and User Specific Data to a client (and for receiving Usage Tracking Data).

Static channel data is simply high-level metadata that describes the channel. Dynamic channel data is used to associate datafeeds with a channel. There can be any number of these records per channel and this information is used to associate those datafeeds with a channel. The datafeed is associated with an adapter and is the mechanism for sending any Custom Channel Data and User Specific Data to a client. Typically, information on the client is added using the IRequestBuffer service and is destined for a given datafeed. Once the datafeed gets that information, it can use it to get any Custom Channel Data and User Specific Data, and return it to the client in a file. This information in the configurator manages this association.

Adding, Modifying, and Deleting Records

Use the following procedures to add, modify, or delete records, using standard Windows techniques to navigate in the Configurator windows. Note that there are two ways to add a record: (1) you can add a new record or (2) you can simply rename or copy an existing record.

To add a record

1. Select New from the Configurators menu.
2. Select a hostname.
3. Select a record type (channels or datafeed).
4. Enter a unique name. (This adds the record to the tree in the left pane.)
5. Configure the record fields as explained below. (If necessary, click Add Record to add additional datafeeds to a channel record.)
6. When done, select Commit from Configurators menu and wait for status message.

To modify a record

1. Right-click on record and choose Rename, Copy, or Delete.
2. Configure the record fields as explained on the following pages. (If necessary, click Add Record to add additional datafeeds to a channel record.)
3. When done, select Commit from Configurators menu and wait for status message.

Channel Configurator

Creating Datafeed Records

Datafeed configuration is closely associated with channel configuration. When an adapter is called, it will generally require input (from the client) to uniquely fulfill the request for data. There are three ways you can do this.

• You can send data directly from client to server using the client's IRequestBufferService.
• You can insert the information from the client in the form of a string into a pre-defined String on the server to make the request unique.
• You can let the client assemble the data it sends through IRequestBufferService into different groups to make channel updates easier.

If you send data directly from the client to the server using the client's IRequestBufferService, the data can sent directly to the adapter with no modification. Any kind of data may be sent this way, e.g. a String, byte array, OFX stream, or serialized object.

In other cases, you might want to insert the information from the client in the form of a string into a pre-defined String on the server to make the request unique. This is done by having a Target Name (see below) defined for the adapter that contains special characters that are replaced with the information received from the client. The most common example of this is a URL string. For example, if there is an adapter that returns a web home page through HTTP and the special delimiting character is *. The target string might be http://www.*.com. Now if the client sent the string mecasw via the IRequestBufferService, the adapter would receive the string http://www.mecasw.com. This facility is particularly useful with the URLAdapter or any adapter using URLs to connect to legacy data.

You can also associate multiple adapters with a single channel to let the client assemble the data it sends through IRequestBufferService into different groups. On the server, these groups are associated with some subset of the adapters associated with that channel. Thus in a single update for a given channel, the client could send several groups of totally different data to the server, which would then be sent to the correct adapter used to fulfill the request.

By using datafeed configuration, you add an adapter and determine how the data it receives will be managed. Part of the function of channel management is to associate this data (through datafeeds) with a channel update request. This assumes that the adapter is an IntelliPortal adapter, or one you developed and put in the outside_classes directory.

Datafeed Configurator

Configuration Examples

The following examples show the process needed to develop some typical adapters. The following high level steps are required to accomplish this, although many of these steps can be done concurrently. See the Channel Developer Guide for a wider discussion of adapter development.

1. Define the desired client functionality. (This is what the end user sees when they subscribe to a IntelliPortal channel.)
2. Define the client and adapter functionality, and what the client and adapter send to each other (strings, serialized objects, arrays of objects, etc.).
3. Write the adapter code (if a pre-built adapter is not adequate).
4. Using the Server Administrator, add the adapter to the server using the datafeed configurator.
5. Using the Server Administrator, add a channel record for the channel with the channel configurator. This associates the datafeed from above with the channel.
6. Write the actual Java code for the channel.
7. Publish and test the channel.

AnswerMyQuestion Channel and Adapter - Simple Example

In this example, the client channel submits a question to the IntelliPortal server, and that question is sent to an adapter for an answer. The response is returned in the file Answer.txt. There are many ways to do this, but for this example, the client will add a String to the IRequestBufferService, and the adapter will be handed this String which it will answer by sending a String back. For simplicity, only two possible questions are asked, "What is the Date and Time?" or "What is your Name?". This completes the first two steps defined above.

Next write the adapter. Normally, an adapter will use the information it received to get information from outside the IntelliPortal server to fulfill the request. In this case, the adapter (com.test.adapters.AnswerQuestionAdapter.java) will only do the following.

public Object process(Object obj) throws AdapterException
{
    String question = (String)obj;
      
    if(question.equals("What is the Date and Time?"))
        return(new String("The Date and Time is " + new Date));
    else // What is your name?
        return("My name is QuestionAnswerAdapter");
}

When done, put the adapter in the outside_classes directory on the server and use the Server Administrator to add it. First we configure the datafeed and adapter. We create a new datafeed record, overriding the defaults, as follows.

Next, create a new channel record, overridng the defaults as follows.

Next, for the channel, we create one new dynamic channel data record, associating the adapter/datafeed with this channel.

This is all you need. The server is ready. If the client, using the IRequestBufferService, does this:

buf.add("What is the Date and Time?");

A file called Answer.txt is returned with the string:

"The Date and Time is Tuesday December 7, 1999 07:21:02".

If the client sends this:

buf.add("What is your Name?");

A file called Answer.txt is returned with the string:

"My name is QuestionAnswerAdapter".

AnswerMyQuestion Channel and Adapter - Complex Example

The first example asked a single question. If the client adds more than one question in a single update, only the answer to the last is returned to the client. This is because the metadata is set up so that the response to the question goes into a single file called Answer.txt. Each time a datafeed returns a response to the plugin, it overwrites the previous file with the contents of the new file. In the next example, we will allow multiple questions. There are several potential ways to do this and the following two examples are discussed:

• One file returned for each answer
  • Set up the metadata so each answer is returned in a separate file. (Several variations are available.)
  • Send multiple questions as a single chunk and have the adapter parse them and send the response in a DataBundle class.
• One file returned with all answers
  • Set up the delimiters so that two questions are added as a single chunk and the adapter parses them and puts all responses in a single file.
  • Instead of adding a single String on the client, add the questions to another object, e.g. a Vector, and send that. The adapter extracts each question from the Object and adds each response to its return String, sending that String when done.
  • Either of the above, but instead of sending the response as a String, send the response as a serialized Object that would be understood by the client and that would house the responses.

One File Returned for Each Answer

The first method only requires a metadata change and one of several naming options that are available. This only shows those fields different than the first example. First, we set up the metadata so that the returned files will be the same as the question asked.

This tells the plugin to use the unique identifier that was sent to the adapter to fulfill the request. The unique identifier sent to the adapter is the question (in the form of a string).

If the client, using the IRequestBufferService, does this:

buf.add("What is the Date and Time?");

A file called What is the Date and Time? is returned with the string:

"The Date and Time is Tuesday December 7, 1999 07:21:02".

If the client sends this:

buf.add("What is your Name?");

A file called What is your Name? is returned with the string:

"My name is QuestionAnswerAdapter".

Here is a variation.

This tells the plugin to use the unique identifier that was sent to the adapter in place of the * character in the Name to map to. The unique identifier is sent to the adapter if the question is in the form of a string.

If the client, using the IRequestBufferService, and does this:

buf.add("What is the Date and Time?");

A file called What is the Date and Time?.txt is returned with the string:

"The Date and Time is Tuesday December 7, 1999 07:21:02".

If the client sends this:

buf.add("What is your Name?");

A file called What is your Name?.txt is returned with the string:

"My name is QuestionAnswerAdapter".

The next option for a multiple file return is to have the adapter return a DataBundle. The DataBundle lets an adapter return any number of distinctly different pieces with one call and return. The client code concatenates each question into one request, and those concatenated questions represent one chunk of data. The questions are parsed by the adapter; each question is answered and that answer is added to the DataBundle, and the DataBundle is returned. The answers are placed in files called Answer0.txt, Answer1.txt, Answer2.txt, etc. Again, only the changes from the initial simple example are shown:

This tells the plugin to replace the # with the index number for that particular piece of data from the adapter. Assume the adapter parses the questions by the ? character. If the client, using the IRequestBufferService, does this:

buf.add("What is the Date and Time? What is your Name?");

A file called Answer0.txt is returned with the string:

"The Date and Time is Tuesday December 7, 1999 07:21:02".

And a file called Answer1.txt is returned with the string:

"My name is QuestionAnswerAdapter".

One File Returned for All Answers

None of these variations requires metadata changes from the initial simple example. All changes are code changes depending on the desired functionality. The coding details are not shown but it demonstrates the versatility of what can be sent through the IntelliPortal server.

First, as in the last example above, the client concatenates multiple questions into one request, and the adapter parses them into separate questions. In this case it answers each question, concatenates the answers, and sends back one response. That response becomes Answer.txt.

In the next example, a serialized object is added on the client, and the adapter de-serializes it, extracts the information, and responds by concatenating the strings together like above. The client code to add the object might look like this:

Vector v = new Vector()
v.addElement("What is the Date and Time?");
v.addElement("What is your Name?");
buf.add(MECAUtil.objectToByteArray(v));

Lastly, the adapter could have inserted the answers in an Object like a Vector, and serialized and returned it, instead of sending concatenated strings. The server would place the serialized Object in the file, and the client would de-serialized it and extract the information accordingly.

AnswerMyQuestion - Using Cache

In previous examples, all requests went directly to the adapter without using cache. The next two examples demonstrate hot to use cache and also how dissimilar pieces of data can be grouped together. Looking at the answers to the possible questions, the answer to "What is the Date and Time?" will always change and can't be cached. The answer to "What is your Name?" never changes and can be cached.

To do this, we set up two different datafeeds, one for the Date and Time, and one for Name. They will both use the same adapter but will manage the data that is returned differently. In the channel record, we will use grouping so the right question goes to the right adapter. The datafeed/adapter records are essentially the same as in the original simple example with some minor changes. Listed below are all modified fields that differ from the default.

This is the adapter/datafeed entry for the DateTime datafeed:

This is the adapter/datafeed entry for the Name datafeed:

Next, create a new channel record. This is exactly the same as the simple example shown earlier. Listed below are the settings to override the defaults:

Next, for the channel, we create one new channel dynamic data record for each of the two datafeeds we want to access. They are similar to the simple example, except that we now use group names and associate them with the appropriate datafeed. First the DateTime record:

Next the Name record:

The questions are now grouped on the client side when making requests. Note that this is a simple example. In a more complex example, any number of chunks of data (e.g. questions) could be sent to each of the groups. Note that since there is no adapter set up as the DEFAULT, if no group name is specified, the request would be discarded once it arrived at the server. Here is the client piece:

If the client, using the IRequestBufferService, does this:

buf.add("What is the Date and Time?", "DATE");

A file called DateTimeAnswer.txt? is returned with the string:

"The DateTime is Tuesday December 7, 1999 07:21:00".

The request would have gone directly to the adapter without using cache.

If the client sends this:

buf.add("What is your Name?", "NAME");

A file called NameAnswer.txt is returned with the string:

"My name is QuestionAnswerAdapter".

If this was the first request to the server for this information, the server would have checked cache, and found that there was no data; then it would have sent the question to the adapter which would have returned the answer. The answer would have been placed in cache so that subsequent requests would have found the result without contacting the adapter.

AnswerMyQuestion - Using Cache with Expiring Data

This example is similar to the cache example above but shows how to set up data that is cached and then expires. To do this, assume that the question is changed to "What is the Date?" (not the date and time). The date is the same for 24 hours, so we set up the datafeed to keep and use the cached data for 24 hours after which it expires and must be re-fetched from the adapter. This means that the adapter would only be hit once per day, for any number of requests from that datafeed for that information.

When data is returned from an adapter, it is associated with a creation time. If the data is set to expire, this time is used to set a start point for determining when the data is stale. By default, if the adapter does not explicitly set a creation time, the time the record was fetched from the adapter is used. In this case, the adapter needs to explicitly set that time.

If the first request for the date arrives at 1:00 AM, the default creation time would not be correct because it would be one hour after the date change; the data would be stale in 23 hours rather than 24. So the adapter would set the creation time to the current time minus the number of total seconds past midnight. This would be done by using the AdapterContext that is available to all adapters and is used to get and set required information. (See the Channel Developer Guide for more details.) The changes would consist of writing the adapter, and minor differences in the adapter/datafeed entry from the above example.

This is the adapter/datafeed entry for the Date datafeed:

All other factors are essentially the same. The same adapter can be used for any of these types of caching, with no prior knowledge of whether or not the data is cached.