Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Scraping Hacker News on a Schedule with TaskPipes (taskpipes.com)
39 points by f292 on Sept 23, 2015 | hide | past | favorite | 12 comments


What Open Source projects are similar? It seems like data transformation tasks like this are a universal problem. I'd just do map/reduce, but that's the advantage that comes with being a developer. Great tools for non-developers to do limited programming tasks are incredibly useful.


Check out Diffbot @ http://www.diffbot.com/ -- you can setup repeat crawls and extract data into .csv/Excel file format or JSON with Crawlbot API, extract data automatically with automatic APIs, or Custom API Toolkit. You can get a free trial account at https://www.diffbot.com/plans/trial to try it out, and you can get paid plans at http://www.diffbot.com/pricing/.


This general category of tool seems very useful. Apparently there are many Yahoo Pipes followers. http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternative...

Support for email and HTML tables seem to be some distinguishing features of this one.


I thought I remember reading that Yahoo was discontinuing the pipes tool?


I've seen something similar more than once on HN but my google-fu is failing. What other similar services are out there that'll let you smartly scrape a page into a spreadsheet.

In the past, I've tried Google spreadsheets with "ImportXML" option but got frustrated after a bit and resorted to python.


Kimonolabs.com

Allows you to scrape then import that data into a google spreadsheet. But its not the only way to consume the data.


Thanks, that's the one I had in mind. Also found : import.io, scrapinghub.com & 80legs.com


Looks pretty but I can't find pricing information.


Chatted with one of the guys

We're starting to move TaskPipes out of beta at the moment, but our pricing will start at $50/month for the service


I was looking for prices as well, or any other indicator for that matter that the tool will be available for a reasonable time at least.


+1

I love these tools but before I invest more of my time leveraging this in my workflow I'd want to know that it'll be around for another few months at least...


I tried NewsBlur as an alternative to Yahoo Pipes. Like it because it has an Android app as well.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: