I know you are using a paid service, I have been using webgrab+Plus to grab data from TV.COM using a cookie, and from tv.sky.com. I am noticing that some channels are from UK and Some are US.I can get horror and syfy from both providers. Last night HBO signature had the wrong movie on. My data was wrong too. I think also I saw a couple streams mixed up last night. Anyway, Some channels are from UK others US. I was using dish as the provider but today I changed it to directtv to compare accuracy. I also see that the data in the XML is changing the icon on the guide in the lower right, usually, using Expatter's file, I get the Channel Icon, in this data pull it shows an icon URL in the file for each show that is on and it changes. I'm sure that increases the size but I don't know how to edit the XML file and the time stamp is whacked!
USING MC2XML TO MAKE YOUR OWN EPG - tuturial with open discussion
- Thread starter expatter
- Start date
You are right about the US and UK streams. What I did initially was to open 2 tabs in a browser: 1 for SKY UK and another for zap2it where I displayed the epg for Directv for Miami. Then I compared what was showing in the websites to what was on the pvr channels. That's how I came up with the difference between which is US and which is UK. If you look at my us.ren and uk.ren files you can see which is which.
Whether using a paid service, Microsoft Legacy, or whatever, you will see that each of them link to a url for logos. I don't think that contributes too much to the file size. At least not enough to really worry about. If you how a low-powered system you could always just decrease the amount of days to download data for.
As for the time stamp being whacked - when you output your webgrab data, make sure it is in UTC, not your local time and you should be good to go.
Give it a try and see if that helps.
Ok I have to say, I re-invented the wheel going with webgrabber+Plus. It has a couple tools in it to like correct the time according the author. I don't see it working but it creates the files. Also it is SLOW!. I have setup several providers. I have separate config files for channels from tv.com, that works half ass. Grabbing data from sky seems reliable in comparison. I get a guide populated ok but I messed with mc2xml and pulled a Microsoft pull for directv satellite. LIke 200mb. I cant do anything with it. I would like an editor like say autoepg or something that would let me look at it. The one with webgrabber works for the sky pulls but not the direct tv us but drop either in kodi and they work. Had to edit a file that contains webgrabb's config data to pull channels each time and it takes 10 minutes. mc2xml did 200mb in no time.
It will work in the native KODI pvr guide. Just like Psyc0n's did. You'd have to go to a forum for those specific addons if you have questions about them.
The 200 MB file sounds like you pulled the provider's entire lineup. So that would also mean that the channel names were not changed to match stalker names and so you would get nothing showing in the guide in KODI.
Since I don't use webgrab I can't give you any help or suggestions with it. Just make sure channel names match what's in stalker and if you have a huge file then your device has to have enough RAM to parse and load the file.
I can only give any help on what I typed in the tutorial. That was the whole purpose of my doing it. If you want to use webgrab I suggest you start another thread on it and maybe someone can help you out.
I totally understand, I figure I will mess with that on my own. Kinda ugly and cumbersome due to time issue only, just my opinion. I pulled working data and its an ugly Beta procedure at this point. I can table it for now.
I think the speed of mc2xml is hands down enough for me to try it. I wasted hours on the other setup. Takes forever to pull and test run from webgrabb and only do like one day of channels and see if it worked. it works.
Now onto what you have done! Great write-up. Ill do a pull of UK data and combine it later. for now I pulled 200 hours from Microsoft. Its about 120MB. Seems too large. I pulled using this :
mc2xml -D US.dat -C US.chl -R US.ren -d 200 -a -U -u -F -o xmltv.xml
I kept getting no available updates or something, so I added force. probably wrong idea, it pulled 120 MB file for the no sports US.ch I made using your tutorial. I just wanna pull one and get it working before combining them.
The fils is 120MB but when I open place it in the path and run Kodi, empty guide. Any thoughts? I mean about this?
Have you created a US.chi file? That is where you limit the channels by either specifying which ones to include or which ones to exclude. If you don't have that file, the pull will include every channel/program listing for whatever data source you specified.
If you don't see anything after the pull, there could be a time issue. The channels/programs are there, but shifted out of the immediate time range and thus not viewable.
If you do forces often enough, they'll block your IP address. The force is considered a no-no and is only to be used if something disastrous happens.
If you don't see anything after the pull, there could be a time issue. The channels/programs are there, but shifted out of the immediate time range and thus not viewable.
If you do forces often enough, they'll block your IP address. The force is considered a no-no and is only to be used if something disastrous happens.
Last edited:
You're totally right about the speed difference between mc2xml and webgrab. And what Ilan said hit the nail on the head, I think. With a pull that large it would have to mean you got their entire lineup. You MUST use a ???.chl and ???.ren file in order to limit the channel data that is downloaded. If you copy my us.chl and us.ren you can always just add the sports channels to them and save a little bit of work.
As far as no new data goes, that is what I was always getting with Dish after the first pull, so I switched to Directv. I think the Dish server has issues cuz even after 3 weeks when I tried to get data from them it still said 'no new data'. I don't have the time, patience or inclination to deal with that sort of nonsense. And I don't want this 'hobby' to become work. And that's exactly what it became today when both of my installs reverted back to the entire NFPS lineup today and I had to take about an hour to put it all back like I had it. Totally blows!
There are still a few channels that refuse to be populated with epg data and I don't know why. But I'm just about ready to totally automate the pulls and write some batch files to accomplish this. For those that wish to do the same, it is explained on the mc2xml website about how to write the files. But they are just text files with the .bat extension. The explanation is in the area about Media Portal or one of the sub categories on awardspace.info.
Remember you can force the file amount to be small by just pulling a day or even less. A day would be -d 24. It's totally easy to combine 2 xml files in mc2sml. just add the -I ???.xml -o guide.xml to your command line.
Aside from notepad++, xml editor is a good way to work with the xml files if you want to view and/or edit them for any reason. There are several other free utililities that will work. Just google 'edit xml files' and you'll see.
I wanted to know if much of this is addressed in this thread
http://iptvtalk.net/showthread.php?2940-NFPS-xmltv-EPG-Grabber-development/page22
I managed to locate a copy of that EPG Grabber, Its is a lot of serious code. Been a long time, however, I managed to get the webgrabber created and my old one backed. I slapped mc2xml in the bin file and launched the batch file and back, it prompted for what type of pull I wanted and chose E for English, bam it running. still waiting...... I guess we will see what the output looks like. OK editing this for the update. shiver me timbers, Pys0ns go bat grabber apparently utilizes some very interesting techniques. It uses both webgrab+ AND mc2xml to pull and consolidate the data, some serious time swapping occurs and boom outputs an NFPFS file. PM me for a copy of this or look for it found it on GitHub. its working. I need to figure out how to change mod the channels, I just edited the *.ch file in mc2xml and will test.
http://iptvtalk.net/showthread.php?2940-NFPS-xmltv-EPG-Grabber-development/page22
I managed to locate a copy of that EPG Grabber, Its is a lot of serious code. Been a long time, however, I managed to get the webgrabber created and my old one backed. I slapped mc2xml in the bin file and launched the batch file and back, it prompted for what type of pull I wanted and chose E for English, bam it running. still waiting...... I guess we will see what the output looks like. OK editing this for the update. shiver me timbers, Pys0ns go bat grabber apparently utilizes some very interesting techniques. It uses both webgrab+ AND mc2xml to pull and consolidate the data, some serious time swapping occurs and boom outputs an NFPFS file. PM me for a copy of this or look for it found it on GitHub. its working. I need to figure out how to change mod the channels, I just edited the *.ch file in mc2xml and will test.
Expatter....Thank you for being so generous as to spend so much of your time "teaching a man to fish". I am an Expat and in the country I am in which is more or less a skidmark in the worlds underwear and I cant even buy USA programming if I wanted to. So for me its a life ring.
Ilan....Thank you for buying me 4 more days of knowing whats up...Without a guide all I am doing is spinning my wheels flipping around and sometimes borking my internet connect which is a combilation of cans, strings, carrier pidgeons and tin foil.
Viper....too few have encouraged you to execute what you intend to do...Hope that you do man. At minimum I say thank you for trying.
FYI..."isreal live" addon did some funky update and self populated pvr simple client today going as far as enabling it. Threw my kodi into fits trying to load prv stalker and kept shutting itself down. Disabled pvr simple 3x and restarted,,,each time it auto re-enabled simple client and re-borked it all. backdoor disabled IL and simple client and the merry go round stopped spinning.....I only mention in case ppl start having same issue and dont realize what is really going on, and start blaming papi or expatters or ilans guide.
Saludos
Ilan....Thank you for buying me 4 more days of knowing whats up...Without a guide all I am doing is spinning my wheels flipping around and sometimes borking my internet connect which is a combilation of cans, strings, carrier pidgeons and tin foil.
Viper....too few have encouraged you to execute what you intend to do...Hope that you do man. At minimum I say thank you for trying.
FYI..."isreal live" addon did some funky update and self populated pvr simple client today going as far as enabling it. Threw my kodi into fits trying to load prv stalker and kept shutting itself down. Disabled pvr simple 3x and restarted,,,each time it auto re-enabled simple client and re-borked it all. backdoor disabled IL and simple client and the merry go round stopped spinning.....I only mention in case ppl start having same issue and dont realize what is really going on, and start blaming papi or expatters or ilans guide.
Saludos
You're welcome.
I long ago stopped letting most addons auto update for similar reasons. I do not like to have someone else take control of my stuff. In fact, I uninstalled most of what I had initially. I found that more addons is just too much and I don't need them since I don't use them. USTVNOW has basic US networks in 720p for free and for $ you can get a whole US lineup in HD. Add stalker and another sub and I'm good. Most of the stuff I get from Specto and don't deal with the commercials.
I ran away from the Israel everything when I first read some horror stories about what ticked off devs do. Ya gotta be careful. Better to not auto update but do it manually after you read in other forums if there have been probs. I learned the hard way bout 4 years ago.
Dang, that stuff seems like ages ago. If I remember correctly, Psyc0n scrapped webgrab due to its being soooooo slow. And messing around with dummy files and all is a PITA (IMO). Though it will give you the wherewithall to reverse engineer and learn the guts of what it takes to make a finished product. That's pretty much how I taught myself to design databases back in the day. Read, copy and modify to suit your own needs. After perusing a few of the pages in that thread, I would say - just stick with mc2xml. And reading all that stuff can make your eyes bleed, so be careful!
Amen, that is about the time that I decided to get this service for my son and I. He loves cartoons and I had to cut the cord when big cable began to bundle everything and hike up ala carte services. I run a Kodi M8s for him and a windows 10 w/ 128GB SSD w /8GB Ram box for whole house and use HDMI to split off to each TV. Mine can handle the 50MB file and probably more.
I see a lot of work on this method here and over at webgrabb+ from both Psyc0n and BlackBear199. Even some of the site INI files there are edited by the same dudes. I see most of the files have no returns in them so the code is smashed up. Hard to decipher it but could use some help. IN that thread Psyc0n said he doesn't want to post go2.bat and does not want people to be taught how to do their own. WOW>
Have you looked at it? Would you look at it?
I see a lot of work on this method here and over at webgrabb+ from both Psyc0n and BlackBear199. Even some of the site INI files there are edited by the same dudes. I see most of the files have no returns in them so the code is smashed up. Hard to decipher it but could use some help. IN that thread Psyc0n said he doesn't want to post go2.bat and does not want people to be taught how to do their own. WOW>
Have you looked at it? Would you look at it?
I've been plugging away at making my own chl and ren files and got a sub with SD. I'm working on plugging some empty channels. I would call myself an "advanced user". Not a programmer. No formal computer education. Using SQLiteSpy I was able to read the TV and EPD DB files from kodi and develop an excel spreadsheet with a channel list, the server channel numbers and stock categories assigned to each channel. I used the thread to figure out how the chl and ren files work. Figuring out where to source each channel has been the biggest pain for me. Since SD limits to 4 threads per day, I plan to try to use free services to plug some channel holes and see if I can create a full english channel xml. I don't even watch most of them, I just hate seeing blanks! I've pulled a bunch of different source xml's to find channels. Thank you expatter for all of your work and tutorial. It would have taken me considerably longer to put it together without your info. Viper8690's guide is great, but I am going to try to finish mine as a learning experience in case something happens along the way with that guide.
Thanks again to everyone that has contributed to this work!
Thanks again to everyone that has contributed to this work!
Just a quick FYI~with Schedules Direct you can have 4 lineups but you can make 6 changes per day. You can use the mc2xml --delete-lineups and when it starts to run the pull, do a control + break and stop the pull. Then you will have 2 more lineups you can try (well, 4 actually).
Thanks. I'm not sure I fully grasp that just yet, but I'll keep going over it. I'll try it and see if that helps me wrap my head around it. There are a few channels that aren't covered by my US, UK and Can pulls. But both you and Viper didn't pull those either, so maybe they are not attainable. Like C-span american history (c-span3).
Interesting. A station like "Press TV" is a 24-hour English language news and documentary network, affiliated with Islamic Republic of Iran Broadcasting. I really don't want a guide, as it is probably something I'll never watch. It would be nice to jog my memory of what it is as I scroll past it. Is it possible to add information so it continually appears in the guide as a description or in the actual time slot for that channel? I also found that Rev'n is an OTA station and has an affiliate in NY. I could create a ren and chl for it. I found C-span 3 from a cox cable provider in Providence RI. Knocking them off little by little, I may be able to fill many of the empty channels. The question is, with SD, how many sources can I pull from? Or should I use microsoft legacy for channels that I do not care that much about and save my SD threads for the others? Again, this is merely a fun exercise in trying create a complete guide rather than something of real importance. I appreciate everyone's time and patience with me. If I get it right, I'd be happy to share any files I have with the group.
At this point you are thinking this is a fun exercise and that's good. At some point you may have a different opinion on it. I am of the opinion that learning something new is a good thing, however my opinion of NFPS is that it is completely arbitrary and follows no logical rules. That's why I have turned off all but 180 channels and will probably whittle that down again since there are many more channels that I have never watched and in all likelihood, never will. To compound the issue is the fact that once you do have a guide and see what should be on and if you click on a channel and something completely different is streaming, you realize that it is in fact, a different channel than the name and logo indicates. There have been channels that I liked to watch that are gone and some channels that just do not work. So for me, this is an exercise in frustration. As such, I am glad that others are stepping up to fill the void. Some users will be appreciative but the vast majority will probably find something to fault no matter what.
To your point of how many sources you can pull from SD, I'm not certain. I've read different viewpoints at several forums in the past couple of years. My thought is that if you draw attention to yourself you will get screwed. I don't think this type of service is quite what they had in mind for end users. Just look what happens when things are publicized on youtube. They disappear fast. Nothing is certain in this game so best to keep that in mind. Here today, gone tomorrow with many. So I wouldn't take it too seriously. Rather just commend yourself on your own accomplishment. As far as sharing...it's nice to share, but once you start down that road people will begin to expect you to support your efforts rather than reward them. And that's where I have a problem with it. Just look at the tone of many posts on this one forum and see that for yourself. So let your conscience be your guide. After expectation comes demands. And this was the whole reason for my post about learning to do for yourself and having a semi-rewarding experience. You rely on only yourself and take solace in that.
To your question about the 'odd' channels - this was handled by using 'dummy' files which were other text files with the names as they appear in NFPS and a ren file that has a generic blurb about what streams on that channel i.e. 'local news and weather for xxxxxxx', etc. Is it worth the bother? Not to me and neither are the channels themselves. So the point here is that 'to each his own' and you simply cannot please everyone, so just please yourself. You said you don't like seeing blank spaces in the guide. My answer is to turn off the channel so it doesn't even appear in the pvr and that is no longer an issue. Do I want to see Indian channels or just about most of the international ones? Nope. So I simply hid them. Period.
I hope this helps you in your decision making process. Have some fun, think of it as a hobby as this service was initially pitched and then pat yourself on the back for whatever you accomplish.
Sorry for the cynical response. I don't mean to turn you off from your good intentions. At the end of the day, there is nothing wrong with a sense of accomplishment. Just be realistic in your expectations.
Cheers.