tv.c File Reference

Read TV Listing xml file from grabtv and build tv_data.gab database. More...

#include <cassert>
#include <cstdio>
#include <cstring>
#include <cctype>
#include <cstdlib>
#include <cerrno>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include "hash.h"
#include "id.h"
#include "data.h"
#include "trace.h"
#include "kbase.h"
#include "dictionary.h"
#include "util.h"
#include "ondisk.h"
#include "calendar.h"
#include "gabbie.h"
#include "tv.h"
#include "time.h"

Classes

struct TvHeader

TvHeader: Data structure at start of file. More...

struct ChannelRecord

ChannelRecord: On-disk record (representation) for a channel. More...

struct Channel

struct ProgramRecord

class IndexNode

Defines

#define TV_ADDR(a)   (((int)a) + sTvData)

#define TV_UNDO_ADDR(a)   (((int)a) - (int)sTvData)

#define CHAN_SIZE   1024

Functions

char * SkipWhiteSpace (char *p)

char * FindStr (char *sub, char *p)

FindStr: Find sub in p, return pointer to where found, or NULL if not.

char StrFxCmp (char *a, char *b)

StrFxCmp: Fixed length strcmp based on length of first string.

void StrCharReplace (char *p, char a, char b)

In string *p, replace any occurence of char a with char b.

void StrReplace (char *src, int srcLen, char *targ, char *repl)

If src contains targ, replace with repl. Src must be large enough.

void AddNumberToIndex (int num, int pos, Id cat)

int IsPunc (char c)

Returns 1 if c is punctuation (except '^' and '-'); returns 0 otherwise.

void RawIndexAdd (Id id, int pos, Id cat)

The actual add, after the processing.

int Exclude (char *p)

void AddToIndex (Id id, int pos, Id cat)

void ReadChannel (FILE *fp, char *p)

char * StrExtract (char *p)

StrExtract: Extract the part of the string between "> <".

void ReadProgram (FILE *fp, char *p)

void UpdateChannel (int key, void *value)

Updates one ChannelRecord.

void WriteIndexRecords (int key, void *value)

void ReadTVListings (char *fileName)

Read an xml file of TV listings created by grabtv.

void PrintProgram (ProgramRecord *pProg, int verbosity=0)

Print a program record.

int FindChannel (Id id)

Find channel indexed by id, and return file offset, or 0 if not found.

int ChanCompare (void *p1, void *p2)

Helper function for sorting channels.

int NewProgCompare (void *p1, void *p2)

int ProgCompare (void *p1, void *p2)

Data * DisplayName (Context *pContext)

Data * QueryPrograms (double startTime, double nowTime, double endTime, Data *pFavChan, Context *p)

Variables

char * sTvData = NULL

Static pointer to mmap'd TV on-disk data.

CalendarDisk * sTvCalendar = NULL

HashDisk * sTvIndex = NULL

int sPrintProgMode = 0

sPrintProgMode == 1, print end times - currently not used

Data * spFavChan = NULL

ChannelRecord * sChanArray [CHAN_SIZE]

Detailed Description

Read TV Listing xml file from grabtv and build tv_data.gab database.

Author:: Bryan Jacobson, 2006.

PLAN:

Phase 1 write to disk:

Credit index (hash table) for credited persons.
Index whole string and individual words. (Example: When is CSI? When is CSI Miami?)
Trim leading "a" or "the" in title and subtitle?
Short titles and subtitles should be stored as names. But, maybe Longer titles and subtitles should be stored as arrays of words. On the other hand, 2 systesm for doing it is not elegant. Maybe an elegant solution is, always use arrays of words, but, if 3 words or less, make a name with those words (minus leading "a" or "the"). Is pollution of name space with huge name strings a problem?

Phase 2 read from disk / process statements/queries:

QueryTV functions
QueryTV grammar rules
Sorted output (sort by date, sort by channel)
Printing channel short "8 KGW NBC" and long "124 NIK Nickelodeon (Pacific)"
Printing program in various ways: Title, Channel+Title, Date+Channel+Title, plus desc, plus rating

Phase 3 Generalized OnDisk function

Abstract out reusable OnDisk class, functions.
May want to move string utilities to util.h (StrCharReplace, StrExtract, etc.)

tv.c Design notes:

The file offset is the unique identifier for instances of channel and program in the TV OnDisk database.

Indices:

ChannelIndex: Find channel by name Id (ABC, KOIN, ESPN) or number (60).
Program: Find programs by name.
Calendar: Finds programs by date/time.
Credit: Finds persons listed in the credit of programs.

TODO: 1. Enhance class Data so it can work in memory and on-disk. TODO: 2. Convert the TV database to class Data (not special structures)

TODO: I'm having a tough time deciding if credits should be indexed separately (index of writers, index of actors), or just Index person->program, and then scan to see if writer, etc. Need a vision for how to index, what to index.

TODO: Need general database creation mechanism - but - kinda need to do a first one to understand how to generalize. Note: Need to look into just using MYSQL.

Data Types:

Listing: One special instance containing general info about this set of TV listings.
Channel: For each channel, an instance with name, number and desc. The Channel inst contains a link to a program on that channel and by following links you can see all programs on that channel.
Program: For each TV program, an instance with info about that program. Programs also contain a link to other programs that start at the same time and other programs on the same channel.
Person: For each person, an instance with a list of prorgrams where that person is credited.
Series: An instance for each recurring TV program.

Define Documentation

#define CHAN_SIZE 1024

#define TV_ADDR ( a ) (((int)a) + sTvData)

#define TV_UNDO_ADDR ( a ) (((int)a) - (int)sTvData)

Function Documentation

void AddNumberToIndex ( int num,

int pos,

Id cat

)

void AddToIndex ( Id id,

int pos,

Id cat

)

int ChanCompare ( void * p1,

void * p2

)

Helper function for sorting channels.

Data* DisplayName ( Context * pContext )

int Exclude ( char * p )

Return true if string is one of "The the And and For for", which are excluded from indexing.

int FindChannel ( Id id )

Find channel indexed by id, and return file offset, or 0 if not found.

char* FindStr ( char * sub,

char * p

)

FindStr: Find sub in p, return pointer to where found, or NULL if not.

int IsPunc ( char c )

Returns 1 if c is punctuation (except '^' and '-'); returns 0 otherwise.

int NewProgCompare ( void * p1,

void * p2

)

void PrintProgram ( ProgramRecord * pProg,

int verbosity = 0

)

Print a program record.

int ProgCompare ( void * p1,

void * p2

)

Data* QueryPrograms ( double startTime,

double nowTime,

double endTime,

Data * pFavChan,

Context * p

)

void RawIndexAdd ( Id id,

int pos,

Id cat

)

The actual add, after the processing.

void ReadChannel ( FILE * fp,

char * p

)

void ReadProgram ( FILE * fp,

char * p

)

ReadProgram() reads an XML record for a program in a TV listing, builds the on-disk ProgramRecord and writes it to disk. ReadProgram() adds entries to indices for this program. ReadProgram() is called by ReadTVListings().

void ReadTVListings ( char * fileName )

Read an xml file of TV listings created by grabtv.
ReadTvListings() reads an xml file of TV listings created by grabtv. This function is called by DoCommand() in gabbie.c. This function calls ReadProgram() to process program information and calls ReadChannel() to process channel information. This function writes out a binary file to DATA/tv_data.gab with channel and program information. The DATA/tv_data.gab file is used by Tv::OpenTvFile().

char* SkipWhiteSpace ( char * p )

void StrCharReplace ( char * p,

char a,

char b

)

In string *p, replace any occurence of char a with char b.

char* StrExtract ( char * p )

StrExtract: Extract the part of the string between "> <".

char StrFxCmp ( char * a,

char * b

)

StrFxCmp: Fixed length strcmp based on length of first string.

void StrReplace ( char * src,

int srcLen,

char * targ,

char * repl

)

If src contains targ, replace with repl. Src must be large enough.

void UpdateChannel ( int key,

void * value

)

Updates one ChannelRecord.

void WriteIndexRecords ( int key,

void * value

)

Variable Documentation

ChannelRecord* sChanArray[CHAN_SIZE]

Data* spFavChan = NULL

int sPrintProgMode = 0

sPrintProgMode == 1, print end times - currently not used

CalendarDisk* sTvCalendar = NULL

char* sTvData = NULL

Static pointer to mmap'd TV on-disk data.

HashDisk* sTvIndex = NULL

Generated on Sun Oct 8 09:32:10 2006 for gabbie by

1.4.6


Classes
struct	TvHeader
	TvHeader: Data structure at start of file. More...
struct	ChannelRecord
	ChannelRecord: On-disk record (representation) for a channel. More...
struct	Channel
struct	ProgramRecord
class	IndexNode
Defines
#define	TV_ADDR(a) (((int)a) + sTvData)
#define	TV_UNDO_ADDR(a) (((int)a) - (int)sTvData)
#define	CHAN_SIZE 1024
Functions
char *	SkipWhiteSpace (char *p)
char *	FindStr (char sub, char p)
	FindStr: Find sub in p, return pointer to where found, or NULL if not.
char	StrFxCmp (char a, char b)
	StrFxCmp: Fixed length strcmp based on length of first string.
void	StrCharReplace (char *p, char a, char b)
	In string *p, replace any occurence of char a with char b.
void	StrReplace (char src, int srcLen, char targ, char *repl)
	If src contains targ, replace with repl. Src must be large enough.
void	AddNumberToIndex (int num, int pos, Id cat)
int	IsPunc (char c)
	Returns 1 if c is punctuation (except '^' and '-'); returns 0 otherwise.
void	RawIndexAdd (Id id, int pos, Id cat)
	The actual add, after the processing.
int	Exclude (char *p)
void	AddToIndex (Id id, int pos, Id cat)
void	ReadChannel (FILE fp, char p)
char *	StrExtract (char *p)
	StrExtract: Extract the part of the string between "> <".
void	ReadProgram (FILE fp, char p)
void	UpdateChannel (int key, void *value)
	Updates one ChannelRecord.
void	WriteIndexRecords (int key, void *value)
void	ReadTVListings (char *fileName)
	Read an xml file of TV listings created by grabtv.
void	PrintProgram (ProgramRecord *pProg, int verbosity=0)
	Print a program record.
int	FindChannel (Id id)
	Find channel indexed by id, and return file offset, or 0 if not found.
int	ChanCompare (void p1, void p2)
	Helper function for sorting channels.
int	NewProgCompare (void p1, void p2)
int	ProgCompare (void p1, void p2)
Data *	DisplayName (Context *pContext)
Data *	QueryPrograms (double startTime, double nowTime, double endTime, Data pFavChan, Context p)
Variables
char *	sTvData = NULL
	Static pointer to mmap'd TV on-disk data.
CalendarDisk *	sTvCalendar = NULL
HashDisk *	sTvIndex = NULL
int	sPrintProgMode = 0
	sPrintProgMode == 1, print end times - currently not used
Data *	spFavChan = NULL
ChannelRecord *	sChanArray [CHAN_SIZE]