Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Is Web Scraping Relevant in Today's Business World?

Different techniques and processes have been created and developed over time to collect and analyze data. Web scraping is one of the processes that have hit the business market recently. It is a great process that offers businesses with vast amounts of data from different sources such as websites and databases.

It is good to clear the air and let people know that data scraping is legal process. The main reason is in this case is because the information or data is already available in the internet. It is important to know that it is not a process of stealing information but rather a process of collecting reliable information. Most people have regarded the technique as unsavory behavior. Their main basis of argument is that with time the process will be over flooded and therefore lead to parity in plagiarism.

We can therefore simply define web scraping as a process of collecting data from a wide variety of different websites and databases. The process can be achieved either manually or by the use of software. The rise of data mining companies has led to more use of the web extraction and web crawling process. Other main functions such companies are to process and analyze the data harvested. One of the important aspects about these companies is that they employ experts. The experts are aware of the viable keywords and also the kind of information which can create usable statistic and also the pages that are worth the effort. Therefore the role of data mining companies is not limited to mining of data but also help their clients be able to identify the various relationships and also build the models.

Some of the common methods of web scraping used include web crawling, text gripping, DOM parsing, and expression matching. The latter process can only be achieved through parsers, HTML pages or even semantic annotation. Therefore there are many different ways of scraping the data but most importantly they work towards the same goal. The main objective of using web scraping service is to retrieve and also compile data contained in databases and websites. This is a must process for a business to remain relevant in the business world.

The main questions asked about web scraping touch on relevance. Is the process relevant in the business world? The answer to this question is yes. The fact that it is employed by large companies in the world and has derived many rewards says it all. It is important to note that many people regarded this technology as a plagiarism tool and others consider it as a useful tool that harvests the data required for the business success.

Using of web scraping process to extract data from the internet for competition analysis is highly recommended. If this is the case, then you must be sure to spot any pattern or trend that can work in a given market.

Source:http://ezinearticles.com/?Is-Web-Scraping-Relevant-in-Todays-Business-World?&id=7091414

Collecting Data With Web Scrapers

There is a large amount of data available only through websites. However, as many people have found out, trying to copy data into a usable database or spreadsheet directly out of a website can be a tiring process. Data entry from internet sources can quickly become cost prohibitive as the required hours add up. Clearly, an automated method for collating information from HTML-based sites can offer huge management cost savings.

Web scrapers are programs that are able to aggregate information from the internet. They are capable of navigating the web, assessing the contents of a site, and then pulling data points and placing them into a structured, working database or spreadsheet. Many companies and services will use programs to web scrape, such as comparing prices, performing online research, or tracking changes to online content.

Let's take a look at how web scrapers can aid data collection and management for a variety of purposes.

Improving On Manual Entry Methods

Using a computer's copy and paste function or simply typing text from a site is extremely inefficient and costly. Web scrapers are able to navigate through a series of websites, make decisions on what is important data, and then copy the info into a structured database, spreadsheet, or other program. Software packages include the ability to record macros by having a user perform a routine once and then have the computer remember and automate those actions. Every user can effectively act as their own programmer to expand the capabilities to process websites. These applications can also interface with databases in order to automatically manage information as it is pulled from a website.

Aggregating Information

There are a number of instances where material stored in websites can be manipulated and stored. For example, a clothing company that is looking to bring their line of apparel to retailers can go online for the contact information of retailers in their area and then present that information to sales personnel to generate leads. Many businesses can perform market research on prices and product availability by analyzing online catalogues.

Data Management

Managing figures and numbers is best done through spreadsheets and databases; however, information on a website formatted with HTML is not readily accessible for such purposes. While websites are excellent for displaying facts and figures, they fall short when they need to be analyzed, sorted, or otherwise manipulated. Ultimately, web scrapers are able to take the output that is intended for display to a person and change it to numbers that can be used by a computer. Furthermore, by automating this process with software applications and macros, entry costs are severely reduced.

This type of data management is also effective at merging different information sources. If a company were to purchase research or statistical information, it could be scraped in order to format the information into a database. This is also highly effective at taking a legacy system's contents and incorporating them into today's systems.

Overall, a web scraper is a cost effective user tool for data manipulation and management.

Source:http://ezinearticles.com/?Collecting-Data-With-Web-Scrapers&id=4223877

How to Trick Google With Your SEO Articles and Web Content

So you're spending time writing SEO articles and creating highly optimised web content, or you're using an article service to create articles for you? What made you click on the link that brought you to this article then?

Perhaps you're looking for a sneaky little trick that will power your articles to the top of the search results in no time at all? You're looking for an edge that no one else has got that will let your content rush to the top of the results like a flatulent cork in water wings? Well read on...

Even the most average internet marketer cannot help but to have become aware that keyword stuffing is no longer effective. Indeed, keyword stuffing is highly likely to see a website demoted or even blacklisted. Today there is a need for high quality content, and for content which is unique and original, as well as popular. The trouble is that this can make the job much harder. Having to spend time creating good, solid, readable content which is useful and interesting is time-consuming.

Having to spend time creating content which might be considered worthwhile by real people is a lengthy an involved process. It used to be so much easier when you could just fling any old rubbish online and let the search engines lap it all up like hungry dogs. Today it seems that those dogs have turned, and unless you want them to bite, you need to spend time actually thinking about your potential customers, rather than just those nice friendly bots and spiders you've been so used to.

This is clearly a difficult situation, and the only option seems to be to succumb to the will of the search engines and spend time creating well-written, highly optimised content that appeals to both the search engines and real people. Goodness - you might even write something people really find interesting, and may want to link to. You never do know these days.

But of course, you clicked the link for this article, because you're looking to change all that. Rather than spending time crafting you'd rather be churning; rather than writing readable content you'd prefer to be chucking out text that looks as though your word processor and your thesaurus have been having an affair!

What you really want is to be able to press a magic button and have your articles fly up the search results, and magically draw thousands of keen, enthusiastic customers flooding to your website, ripping open their purses and wallets with such feverish excitement that you'll hardly know what to do with all that easy cash you'll be wallowing in.

As someone who provides an article service to internet marketers and business owners, and who writes SEO articles for a living, I have a few words of advice for those of you who want to try to get your articles above mine, who want to see your articles power ahead of mine and take hold of the search results pages by the horns.

Whilst I may sit here taking time to research each and every article I write, plan every article so that it has something to say, write it in a way that makes it entertaining, enjoyable and informative for those real live people who exist out there on the other side of the web, craft articles in a way that takes full advantage of Google's algorithms, optimised for latent semantic indexing, yet making it almost entirely undetectable, you want to discover a secret formula that will launch your articles with barely more than a flick of your wrist.

You probably want to find out what this secret formula is so that you can spend less time hurling hundreds, perhaps thousands of articles out every week just to scrape by. Meanwhile, I'll write an article once every week or so. You'll notice them because they always end up boosting my website up to the very top of Google for all the major keywords and key phrases I have chosen, despite several billion other sites all appearing for the same searches.

Well, here it is. The magic formula, the button you want to press is coming right up. Forget those black hat techniques that simply blast meaningless content at thousands of identical directories. To really achieve success with your SEO articles and enjoy the same level of exposure as my article service, the magic formula is this: forget writing SEO articles. That's it. When you're writing your next article, forget that it's an SEO article.

Source:http://ezinearticles.com/?How-to-Trick-Google-With-Your-SEO-Articles-and-Web-Content&id=4078570

Restaurant Internet Marketing - It's Not Your Father's Yellow Pages

As a restaurant owner or manager are you taking advantage of all the opportunities offered up by the Internet or are you still counting on word of mouth and a traditional print marketing program to get diners filling your place? Are you even aware of what the Internet marketing can do for you? If you're not you're giving up business to your competition.

Take a moment to Google your type of restaurant (Italian, seafood, deli etc) and the name of your town and see what comes up on the first page. Do you see your name? How many competitors do you see? Now ask yourself "How many potential diners are going to come to my restaurant based on these results"? The answer of course is nada. Can you afford to give that business to your competition?

Location, location, location

Many local small and medium sized businesses have discovered the value of having an internet presence but it wasn't always that way. Most local businesses thought the internet best served big companies. On top of that, most small local businesses simply didn't have a clue how to participate effectively and they were too busy running their businesses to learn how.

The fact of the matter is that 72% of all searches are related to a search for local content. Nearly a year ago the monster search engine Google changed their algorithms so that local results show up on a search inquiry whenever appropriate. That means your shop can be just as competitive as a national restaurant chain when it comes to being found in search.

The difference between search and traditional advertising

So why is search such a big deal? The reason it is so much more effective than traditional marketing is that it responds to a specific need at a time when the potential diner is interested in the information. Unlike a weekly flier that's dropped in the home mailbox every Thursday, search doesn't clutter up a prospective diner's life but rater provides relevant information when the diner wants it not when the USPS delivers it.

When that potential customer is thinking about where to go for lunch he or she is going to search "pizza san pedro, ca" to find what's available in the area. In other words search delivers the information when the searcher is most receptive to it.

But wait there's more

Search by itself is reason to have a presence on the internet but the benefits certainly don't end there.

How would you like to be able to:

- Notify all of your customers what your daily specials are every day automatically.
- Invite your customers to place their order in advance so they don't have to wait for their food and do it without spending any time on the phone.
- Accept and confirm reservations electronically.
- Run promotional campaigns without spending a dime on advertising.
- Build solid customer relationships that will strengthen your word of mouth referrals.

And that's just the beginning. A professional internet presence will cost you significantly less than you are currently paying today for marketing. What's more, professionally managed internet presence keeps on giving even when you stop paying. Can you say that about your Yellow Page listing or print ads?

Online Reputation Management

There can be a potential dark side to this whole internet thing. You live or die on your reputation and it is important that you manage that reputation online. Never has the consumer had such an easy way to post anonymous comments for all the world to see. If you have a customer who has a bad experience don't be surprised if he or she posts it to a blog or a forum or gives you a bad review in the business directories.

Speaking of business directories, you already are on the internet even if you didn't know it. These directories scrape information out of Yellow Pages and other sources and add it to a business profile. You have a right to claim those listings and they are actually a great way to promote the business. But registered visitors also have the right to post reviews and if the only review your store has is from that disgruntled customer...your rep is shot.

There are just too many reasons that you have to get involved with the internet marketing. To not do so is to pass up an incredible opportunity for new business and to cement loyalty with regular customers. It's effective, inexpensive and essential if you want to remain competitive.

WebFrootz is an internet marketing agency, Baltimore website design, and Baltimore web development company MD specializing in comprehensive marketing solutions for small and medium business including branding, Website design and development, graphic design, copywriting, and SEO.

Source: http://ezinearticles.com/?Restaurant-Internet-Marketing---Its-Not-Your-Fathers-Yellow-Pages&id=6498349

Infinity Logo Design Launches Content Writing Service

Infinity Logo Design recently added Content Writing Service to its pool of services, available at a staggering discount of 70%. Following the huge success of its Animated Video Service, Infinity Logo Design has now officially commenced its Content Writing Service as well. With a clientele of nearly a million, Infinity Logo Design is one of the fastest growing design companies of the world. And its recent diversification advocates the fact that the company looks fanatical in achieving its goals and objects with one of them being "to become the number 1 design & development company of the world". Providing a complete range of Content Writing services which includes Web Content Writing, SEO Content Writing, Creative writing, Press Release writing, Article Writing and Brand/Product Name Suggestions, Infinity Logo Design is dedicated to providing a superior experience to the readers.

According to the company's Creative Manager, "We have been offering Content Writing service as a secondary service for quite some time, but following its augmented demand and some great feedback from our clients, we decided to launch it officially. We consider Content Writing as an art that truly fascinates and inspires the readers, and makes them believe in the message that the writer is trying to covey. Our copy writers are handpicked, and have a proven track record. We provide our customers with nothing but the very best!"

Infinity Logo Design believes that Content Writing in an effective tool for customer retention and generating a profound user experience. Infinity Logo Design is offering this service at quite a substantial price, and its exceptional discount of 70% is a bonus for all those who are in pursuit of quality content at an affordable price. Judging from the company's recent endeavors, there's no denying that we'll be seeing some great things from them in the years to come.

About Infinity Logo Design:

Infinity Logo Design is one of the world's leading design companies. The company offers a wide range of services including custom logo design, website design, brochure design, banner design, stationery design and promotional items. ILD provides high-quality designs to small- and medium-sized businesses at the most competitive prices. In addition, the company engages the services of skilled and talented designers who can create your design exactly according to your specifications. The Company also offers 100% satisfaction and 100% money back guarantee, making it one of the most sought-after design companies in the world.

Source: http://www.digitaljournal.com/pr/1985193

Why I Use an Anonymous Web Proxy Service

I've been using an anonymous web proxy service for some time now and I don't think I would ever go back to ordinary surfing.    It's not that I am some sort of shadowy cyber criminal with some secret online life, no I pretty much surf the same web sites as everyone else, I certainly don't go to illegal or criminal sites.

If you want to know why, it's simply because I value my privacy and I know what data is stored in the average organisation and how it is protected.   Take your ISP for example, they logs pretty much everything that passes through them - every single request, every web page, every email and even every image is logged.   The vast majority of this traffic is in cleartext so the majority of it is instantly readable and can be matched to your address. 

It's amazing what these logs can tell about people you can build up a lot of information about someone from what they do on the web.  So do you know who has access to these logs at your ISP, what controls are in place, how is this information stored ?

If you don't know your not alone but after 25 years working in IT, I'll pretty much bet you wouldn't like to know the answer anyway.  This is the problem with much internet communication, it's so completely open - occasionally you'll obviously use a secure site to type in your credit card details.   Yet up to the point all your information is flying across the internet ether completely unprotected and accessible to anyone who has the will and the knowledge to intercept it. 

Believe me there are plenty of people who do have an interest in your information

Why do you think a European Directive was recently passed - Directive on Mandatory Retention of Communications Traffic Data.   It's  a bit of a mouthful but what it's  actually doing  is forcing your ISP to keep a record of every email sent, every internet session and web site visited for two years.   So think back at every web site you've ever visited for the last two years and it's on record and can be matched to your own little electronic profile.

How does that make you feel?

Even if you've done nothing wrong it's bound to make you feel a little uneasy after all why do Governments want all this information about us?   If you want to feel a little more uneasy, many Governments are looking at the UK Governments idea who are going to store all this information in a central database accessible to certain organisations.     Fear not US citizens you'll not be left out - the FBI are pushing hard for similar data retention facilities.

Storing data is dangerous

If you store data you have to be prepared to take good care of it.   All the privacy erosion concerns aside, the more personal data is stored about us, the more people will have access to it.  How thorough do you think is the vetting procedure for an ISP Technical engineer ?  Who potentially has access to huge amounts of this data.    Even if implicitly trust our govenrments with this tremendous invasion of our privacy - fraudsters, identity thieves and others can create mayhem with this sort of information. 

These logs are gold dust to anyone trying to steal our identity for example - they can match up users, with web sites, pick up password and all sorts of personal information.  Even  though a small part of this information is encrypted when you're actually using a secure web site (with the padlock below) there's plenty of other personal information available to compromise your usernames, password etc.

An identity thief can pick up huge amounts of data about the average person all ready, an hour or two picking up all your web traffic and they'll also have where you bank, what web sites you visit and logon to and a whole heap more.  This unfortunately is the tip of the iceberg and I suspect you'll see many, many people protecting  their privacy online soon from identity thieves, hackers and even our friendly snooping governments.

Everything I do online is private, I use a fast professional service that anonymises my connection and encrypts all the data.  My web logs sit next to yours in a server room at my ISP however mine are all completely encrypted and unreadable by anyone, whereas yours are in clear text.

One thing I should warn people about is the use of free anonymous web proxy services which you see all over the web.   Be very, very careful about these and think why are they supplying a very expensive and resource hungry service for free?   Remember also that by using one of these free proxies you are redirecting your traffic via this server whose owner can capture and log your data himself.   Many of these are owned and run by Eastern European Hacking groups as an easy way to gain peoples traffic, so be careful.

Source:http://ezinearticles.com/?Why-I-Use-an-Anonymous-Web-Proxy-Service&id=1679289