ccs

CCS - Clearing Center Server

Introduction
File conversion
File transmission
Application watchdog
(Current state of CCS) outdated
To get the current state look at the History of CCS
download source code

Introduction

CCS will be an integrated system to manage EDI, file transmission, file conversion and external applications. It will be available under the GPL and run on Linux and other Unix flavours.

The main part will be the ccsd, a daemon consisting of several processes running on a trusted server. ccsd takes care of incoming files, processes them as defined in project files and delivers the output to the appropriate application. All file transfers are logged in a global history. Another part of ccsd will periodically check the availability of other applications.

File conversion

For converting data from one file format to another there is usually a bunch of converting utilities written especially for this purpose. CCS has a different approach : There is one file format, CCS uses internally, hds, the hierarchical data structure. This format is built as a tree representation of data elements, each element may have one follower and one child. An element is defined with a key, a name, a type, a maximum length and the value. For handling hds files there is the libhds with all functions necessary to create, manipulate, read and write hds files.

A new files format alone is - of course - not the solution. hds has support for file format plug-ins, which offer two functions : one for reading a file format into a hds and one for writing a hds in that format.

To convert files from one format to another there are the following steps to take :

  • read the source into a hds
  • read a description of the destination file into a hds, this creates a skeletton of the output file
  • map data elements from source to destination hds
  • write the destination hds using the destination file format

    The mapping is done once per application with a GUI (khds) which creates a mapping file. This is read by ccmapper and processed from top to bottom.

    CCS uses several helper functions packed into libraries :

  • libhds, hds handling functions and format handler via plug-ins
  • libstrtool, functions to manipulate strings more easily, e.g. splitting word lists, table handling
  • libexpr, evaluation of conditional and arithmatical expressions as well as simple code conversions by looking up in conversion tables
  • ccmapper, the batch file converter, executable using above libraries

    File transmission

    The most important feature of a clearing center software is IMHO the handling of data flow. You should be absolutely sure that no files are lost, no delay occurs without a notification and the possibility to track a data file through all the steps it takes through the clearing center. When an error occurs there should be a verbose error log that tells you what has happened. The state of each data file is saved and may be viewed and manipulated. After a failure the server must be able to continue with the step that didnīt work.

    To achieve that, several functions have to be implemented :

  • a way to copy files from one location to another, errors are reported as verbose as possible
  • each file transmission is logged, each step which has to be done and a report of failure or success of that step, this is necessary for the server to restart a job in case of a failure.
  • known file processing times for the server to detect anomalies in receiving files (may be a broken input link) and broken applications supplied by CCS

    Application watchdog

    This goes hand in hand with the file transmission. As CCS will not be able to do all the necessary work in your clearing center there will be other applications, such as EDIFACT converters, data base links and more. CCS should know how to check these applications whether they are still operative. In case of failures it would be able to inform you. The detection may include the following methods:
  • an application has not sent files for a period of time
  • an application was supplied with a file from CCS and it hasnīt returned it in time
  • trying to access a service from this application fails

    In CCS you will be able to define check methods, that can be run periodically or when you (or CCS) expects an error. This process will start with a simple check, e.g. a ping to the server. Depending on the success of the first step another check will be performed or a failure reported. These methods will have access to the file transmission history to check for the last successful action and more.

    Current state of CCS (by Jan 24th, 1999)

    Read the CCS History for more up to date information. This will stay here because I do not want to keep this text current for all small changes.

    First of all, ccsd isnīt yet. I started with little things around CCS but it is not an integrated system yet.
    libhds is ready, as far as I can say now, libstrtool and libexpr are usable and will be enhanced when neccessary.
    ccmapper works for simple conversions, but it is not that flexible as planned, more work has to be done on that. The GUI part, khds grows and finally got the possiblity to create project files. Some aspects of mapping, especially the different handling of repeated structures, is not solved well in ccmapper. Iīm still searching a more flexible way to define that.

    For the application supervision I had something working, but that was lost on a broken disk (Do backups, believe me!), but I still have it in mind how to do that.

    For file transmission there are some functions for interpreting URLs, reading and writing mainly and a test program to copy data from one URL to another. Currently ftp: and file: is supported. In case something goes wrong, you get an error code and the log of the ftp session back.

    As the source code is much spread apart I have to reorder and prepare it before releasing it.

    There is no logo yet ;-)


    CCS draft, January, 24th 1999,©1999 Rolf Jakob rjakob@duffy1.franken.de
    Last change : February 16th, 1999