Google Refine lets you fix and handle huge, messy sets of data

By Erez Zukerman, Download SquadNovember 17, 2010 at 10:30AM

Filed under: ,

googlerefine

Google has just introduced a new product, and this time it’s a PC application (with a browser-based UI). It’s called Google Refine, and it solves a problem that is enormous for some people: it lets you take massive sets of “messy data” and massage them into shape so that they’re uniform, make sense, and can be statistically analyzed.

The video after the jump shows a very good example, which is based on a CSV file exported from a publicly available data source (a government contract system, in this case). The data is very realistic – descriptions are inconsistent (Firm Fixed Price on some rows and FFP on other rows), and even the number formats are inconsistent (you get 0.78 on one row and a number in the millions on another row).

Google Refine lets you very easily hone in on those inconsistencies and fix them in a myriad of ways. This is an important data tool because those heaps of messy data are often public records, which are available but not transparent; being able to quickly analyze them could expose some very interesting patterns and anomalies in the way that public institutions and governments behave.

[Thanks, Yanksy, for the tip!]

Google Refine lets you fix and handle huge, messy sets of data originally appeared on Download Squad on Wed, 17 Nov 2010 10:30:00 EST. Please see our terms for use of feeds.

Read | Permalink | Email this | Comments

Add to digg
Add to del.icio.us
Add to Google
Add to StumbleUpon
Add to Facebook
Add to Reddit
Add to Technorati


Process Explorer v14 Better Monitors Chrome and Other Multi-Process Apps [Downloads]

By Kevin Purdy, LifehackerNovember 17, 2010 at 08:30AM

Process Explorer v14 Better Monitors Chrome and Other Multi-Process AppsWindows only: Process Monitor, our preferred Task Manager replacement, has updated with a host of new features and improvements. What you’ll most notice is a new column that can provide aggregate CPU usages of multi-process apps like Chrome.

To check out the new column, right-click in Process Explorer and choose “Select Columns.” From the “Process Performance” tab, select “Tree CPU Usage.” You’ll now see the total processor pull of apps like Chrome that use multiple processes to do their work. Other improvements to Process Explorer include built-in disk and network monitoring, more memory statistics, more reliable DLL scanning, and other goodies outlined at the Sysinternals blog.

Process Explorer is a free download for Windows systems only.

Process Explorer v14 [Microsoft/Sysinternals via Ghacks]

Using purge to free inactive memory on Mac OS X

By (author unknown), Chris Hope's LAMP BlogNovember 16, 2010 at 11:00PM

Apple Mac OS X is generally good at memory management but I sometimes find the inactive memory takes a long time to be released as free memory and my system can grind to a bit of a halt if I need to start up a virtual machine. This post shows how to use the purge command to free up the inactive memory.

Free, Wired, Active, Inactive, Used Memory

The screenshot below is from the Activity Monitor and shows the current free, wired, active, inactive and used memory.

memory usage in mac osx activity monitor

Briefly, free memory is available to be used right now; wired memory is basically system memory for the kernel and other stuff; active memory is memory being used right now; and inactive memory stores info from recently quit applications. The idea with inactive memory is that if you quit and app and start it up again it’s going to launch nice and quick because the info is still in RAM.

Inactive memory can be slow to free up

One of the issues I’ve often found is I want to fire up a virtual machine, which needs say 600MB or so, and I’ve only got 350MB or so available free but loads of inactive memory, as shown in the screenshot above.

I start the virtual machine up and then may as well go and have a lie down while I wait. The free memory quickly gets used up and then slowly, ever so slowly, memory gets freed up from inactive.

Sometimes all you need to do is quit a few apps and it sorts the issue out, but sometimes I’ve done this and I still don’t get much free memory made available, or I need to keep using the other memory consuming apps while I debug something in a web browser in a VM.

Using the purge command

There is a command called “purge” which can be used to free up memory. You should really use it sparingly (if at all) but it does actually free up all the inactive memory. Note that I’m not necessarily recommending you use this function but that is available should you want it.

Simply open up a terminal and run this:

purge

It may take a minute or so to free up the memory, and you may find things run a little slowly for a bit but sometimes if you really need that memory freed up right now it’s the only way I’ve found out to do it.

I’ve generally found it’s faster for me to use purge before firing up a VM if I don’t have enough free memory available than wait for the system to grind along while the VM boots up if a lot of RAM is inactive.

All-optical transistor created

By (author unknown), Gizmag Emerging Technology MagazineNovember 16, 2010 at 06:26PM

The MPQ/EPFL microresonator, which couples light with vibrations (Photo: EPFL)

Researchers from Germany’s Max Planck Institute of Quantum Optics (MPQ) and the Swiss Ecole Polytechnique Federale de Lausanne (EPFL) have created a microresonator that produces vibrations from laser light. The device also uses one laser beam to control the intensity of another, thus making it essentially an optical transistor. The technology could have big implications in fields such as telecommunications.
..
Continue Reading All-optical transistor created

Tags: ,
,
,
,

Related Articles:

How Can I Find Out If My ISP Is Limiting My Download Speed? [Ask Lifehacker]

By Amit Agarwal, LifehackerNovember 16, 2010 at 02:00PM

How Can I Find Out If My ISP Is Limiting My Download Speed?Dear Lifehacker,
Recently, I’ve felt that my download speeds have been a little uneven, and I’m starting to wonder if my ISP is limiting or throttling my downloads. How can I tell for sure?

Signed,
Up for Downloading

Dear Up,
We’ve covered this topic once or twice in the past, but it never hurts to get a refresher. For that, let’s turn to our pal Amit from Digital Inspiration:

The following is republished with permission from Digital Inspiration.

You have a fairly good Internet connection at home and regular web sites load pretty quickly in your browser. However, the speed seems to go down while you are watching videos on YouTube or are trying to download files through a torrent client.

If you observe such a speed pattern quite frequently, chances are that your ISP could be rate limiting your traffic for certain bandwidth intensive operations. To give you an example, if your regular download speed is 100 kB/s, YouTube videos could be streaming at a speed of 30 kB/s due to rate limiting by the ISP.

Is your ISP is limiting your download speeds?

You can run the Glasnost test in your browser to determine whether or not your ISP is following any such tactic to manipulate your download speeds for specific sites.

How Can I Find Out If My ISP Is Limiting My Download Speed?

The test uses a Java applet to compare your regular download speed against the speed at which Flash videos get streamed to your system. Other than videos, it can also compare the download speed for email attachments (via POP and IMAP), normal HTTP based file transfers, torrents, and binary downloads from Usenet servers.

You should consider running these tests at different times of the day since some ISPs may be limiting speeds only during peak hours. Also, do remember to stop any other downloads that might be running in the background for more accurate results.

Here are some additional tips on troubleshooting a slow connection.


Thanks Amit, and thanks for the question!

Love,
Lifehacker

P.S. Have a preferred tool or method for checking to see if your connection’s being throttled? Let’s hear it in the comments.

Amit Agarwal is a personal technology columnist and founder of Digital Inspiration, one of the most widely read how-to blogs in the world.

16 Months Later, Say Hello To Google Voice For The iPhone. It Was Worth The Wait.

By MG Siegler, TechCrunchNovember 16, 2010 at 01:50PM

Over the past few years, there has been no shortage of developers complaining about how long Apple has taken to approve their iPhone applications. But few have waited as long as Google for the Google Voice application. But today brings great news: the wait is over.

Yes, the app is real, it’s approved, and it should be live at this link. The official Google Voice for the iPhone is here. And it’s wonderful. Yes, it was even worth that insane wait.

As you’re probably well aware, this app has had quite the contentious history. Google originally submitted it to the App Store about a year and a half ago. But 16 months ago, Apple decided they weren’t going to accept it. Not only that, other third-party apps with Google Voice functionality were pulled from the App Store as well.

While Apple and Google were both silent on the issue at first, the government got involved and both sides were made to disclose at least part of their stance on the app, their relationships with carriers, and the relationship with each other.

Apple’s stated stance was that they didn’t like the Google Voice app because it was too similar to the functionality of their own built-in calling system on the iPhone. They also said that they hadn’t rejected the app, just that they were still considering it for approval.

BS or not, Apple controlled their own App Store, and there was little Google could do. The decided to build a web app for Google Voice instead.

But this past September, Apple, perhaps pushed by the governmental inquiries, decided to ease up on many of their strict App Store rules. This led to third-party Google Voice apps being allowed back into the App Store.

And while it took a little longer than expected, now we have the official app as well. Unlike the third-party versions, Google’s app is free. It’s also quite a bit faster thanks to APIs that Google hasn’t made publicly available just yet. The result is a great app that is likely to be many user’s go-to dialing and message management service on the iPhone.

If you’ve used Google Voice on your computer or other devices, the iPhone version should be familiar to you. However, unlike the Android version, Google Voice cannot totally take over the calling capabilities on the iPhone. All that means is that you have to open the app instead of your regular iPhone Phone app to use it.

But you can make calls right from the app. You dial a number and it automatically takes you to the iPhone’s Phone app where it places the call from your Google Voice number by way of Google’s backend system.

When you receive a call to your Google Voice number, it will come in to your regular Phone app as well, just as happens now with call forwarding.

You can also send and receive texts from the app. And you can listen to your Google Voice voicemails — and yes, all of them are transcribed for you to read in the app. And all of this works with the iPhone’s Push Notification system.

And unlike the HTML5 version, the native app can access your phone’s contact list. This make it easy to place calls and set up Quick Dial Contacts.

Google Voice remains U.S.-only for now, but the team says it will roll-out internationally in 2011. It is also iPhone-only for now. Though Google envisions both iPad and iPod touch support in the future. That said, Google Voice is still not a VoIP service, so you won’t be able to make calls without service. The app requires iOS 3.1 or later.

Again, you can find the app in the App Store here. It’s a free download.

Update: And now you can read more about it on Google’s blog.

Tale of Infamy: Skunk Funk

By Sean O’Hara, ToolmongerNovember 15, 2010 at 02:28PM

Just as the weekend began, a skunk sprayed my dog. The five minutes that followed that event could have been written into any comedy you’ve seen in the theater. However, when it happens to you, it’s less funny and more irritating than anything else. In short, the dog bolted in the house full of skunk spray and proceeded to rub and shake it all over the house before I could catch him. It was great.

So let’s just pretend for a minute that you’re standing in your living room with a wailing dog in the laundry room full of skunk smell, a screaming baby in the bedroom, and a second howling dog in the kitchen who is just making noise because everyone else is making noise. The damage is now done. It’s time to start fixing what has obviously gone horribly wrong.

The first step is to make sure everyone is okay. In my case the baby and second dog were just yelling to yell, and because it smelled bad. Talat, on the other hand, had real problems. Skunk spray will bond to any fibrous surface like hair (which dogs have a lot of) quickly. So it’s very important to get as much off as fast as you can. He got hit square in the face at what must have been point blank, because his entire muzzle was yellow and his eyes were matted shut.

It hurt like hell, I’m sure, and the spray can temporarily blind dogs and people, so into the tub he went with a bottle of ketchup, a sponge, and pitcher. The trick here is to dilute the spray as much as you can and get it away from his eyes. Then you need to stop it from spreading — hence the tub. Scrub and wash, rotating between shampoo and ketchup until you’ve got the worst of it off. You can also use the formula I will outline in a minute, but it will sting worse than the spray itself if it gets in the eyes.

Once the worst is off the dog and he is safely put in the garage to dry off, it’s time to survey the rest of the damage. Everything in your house will stink badly if even just a little spray winds up in your place. There is a solution you can find on the ‘net that is said to help with the defunking of smells, so I called upon it, hoping its mystic homebrew goodness would save my olfactory senses.

It consists of:

1 gallon water
1 cup baking soda
1 tbsp dish soap
1 qt hydrogen peroxide

Mix it all together, load it into a spray bottle and start hosing your place down. It will not remove all the smell but it will start to give you and your family relief from the stink. Spray effected areas — which is everything — as many times as you think you can handle. It will get better.

Contrary to popular belief, you are not done. Wash the animal in the solution with a sponge to get the rest of the smell out of his furry coat. It works great on dog hair, and Talat was the best-smelling thing in the whole house after his second bath.

You should at least be able to sleep in the house the first night at this point. Most likely 70 percent of the smell is gone and everyone is no longer gagging if they enter the house. The next morning (or that same day if it happens early enough), get a hold of a steam cleaner and go to town. Clean everything you can with it.

Next sprinkle baking soda all over the carpeted areas, fabric covered furniture etc., and let it sit for about a half hour. Then grab a vacuum and pull it all back up.

If it’s at all possible, open the windows to get some fresh air coming through the house. It will help as much as anything else. Smaller things like changing the furnace filter and washing all your clothes will also assist in the exodus of funk.

Three days later I can say the house is mostly good to go. There is a faint smell if you know where to stand, but the harshness is gone. Repeated attempts at the steps above will also continue to clear what remains. Time will eventually clear the rest of the horrid order from its places of hiding, but don’t expect wine and roses immediately.

[Note:] The best solution by far would be to keep the damn dog away from skunks.

Katana v2 (y0jimb0) – Portable Multi-Boot Security Suite

By Darknet, Darknet – The DarksideNovember 15, 2010 at 04:53AM

Katana is a portable multi-boot security suite which brings together many of today’s best security distributions and portable applications to run off a single Flash Drive. It includes distributions which focus on Pen-Testing, Auditing, Forensics, System Recovery, Network Analysis, and Malware Removal. Katana also comes with over 100 portable…

Read the full post at darknet.org.uk

Know Your Rights; Ten Misconceptions About Photography Law [Photography]

By Jason Fitzpatrick, LifehackerNovember 12, 2010 at 09:00AM

Know Your Rights; Ten Misconceptions About Photography LawBecause most people learn about photography laws and photographer’s rights by word of mouth—and often from other photographers—there is a lot of room for misunderstanding. Carolyn Wright, an attorney specializing in photography law, clears up ten common misconceptions.

Photo by Rocco Lucia.

Wedding Photographers International, a trade group for wedding and portrait photographers, interviews Carolyn Wright, an avid photographer and attorney specializing in photography law. She debunks ten commonly held misconceptions of photography and the law. Among the most widespread of misconceptions is that anything in view of the public is fair game:

You don’t need permission to photograph a work of art that is in a public area.

This rule is based on copyright law. United States Copyright Law grants exclusive rights to the copyright owner of a creative work, including the rights to: reproduce the copyrighted work; prepare derivative works based on the copyrighted work; distribute copies of the copyrighted work to the public; and/or display the image. (See 17 USC §106.)

When those rights are infringed the copyright owner is entitled to recover damages suffered as a result of the infringement. (See 17 USC §504). So even when a creative work is in a public area you may photograph it only if the work is in the public domain or your photograph makes a fair use of the work.

For more common misconceptions—and a better idea of what constitutes the sometimes nebulous “fair work” clause in copyright law—check out the full list a the link below. For more articles on photography and the law, check out Know Your Rights as a Photographer and the Ten Legal Commandments of Photography.

Top Ten Misconceptions about Photography and the Law [Wedding Photographers International]