Stack Exchange
log in sign up chat discuss faq users badges

Area 51 » Technology

definition commitment beta
331
followers

Big Data

Q&A site for administrators, developers, analysts, and statisticians dealing with enormous data sets

11
votes
1
answer

Have you added five questions? Have you used all five of your upvotes on questions with less than 10 votes?

jan 9 '17 at 18:31 Robert Cartaino 473
3
votes
0
answers

Questions of closed area51 proposals

jun 6 '14 at 4:25 Handsome Nerd 236
4
votes
0
answers

Alternate Proposal

oct 22 '13 at 14:12 leeand00 1,734
3
votes
1
answer

Does 'big data' also include 'large, complex data'?

may 15 '13 at 12:25 Community♦ 1
14
votes
1
answer

Duplicate of closed proposal - does that make sense?

may 10 '13 at 16:06 Community♦ 1

show 6 more discussions
discuss this proposal

49 Example Questions (1 closed)

active newest votes
up vote 20 down vote
How do I anonymize large data sets efficiently?
added by Oleksi Jun 15 '12 at 2:14
link
6  
Votes over 10 for a particular question don't help to create the community. Rather than vote again for a popular question, it's best to look at some of the questions with fewer than 10 votes and upvote them. Don't forget that you can also move your votes by clicking the up arrow on a question you've already voted for to remove your vote, then voting again on another question. – MCeley Jan 25 '13 at 13:29
1  
14 people who like this question could move their votes to other questions with below 10 votes. Scores over 10 don't help. We only need 6 more example questions with a score of 10 to go to beta! – Duncan Feb 13 '13 at 0:19
up vote 17 down vote
What data structures can be used to handle large amounts of data efficiently?
added by Oleksi Jun 13 '12 at 5:13
link
6  
Too broad perhaps? – Tibor Aug 6 '12 at 13:42
2  
I think a good version of this question would have to also include the uses cases in the actual question body. – Oleksi Aug 21 '12 at 17:08
1  
9 people who like this question could move their votes to other questions with below 10 votes. – Duncan Feb 13 '13 at 0:20
up vote 16 down vote
What software is available for managing large volumes of data?
added by Joe Jun 28 '12 at 13:48
link
2  
Isn't this a shopping question and therefore too subjective for the normal Stack Exchange Q&A structure? – RobM Dec 13 '12 at 19:29
@RobM : perhaps, but I've had quite a few long discussions in various places about how the only real difference between most questions in Stack Overflow and Server Fault is that being software, they can be tested and verified in short order. I'd say that this being one of the higher rated questions is a sign that we need a forum for asking this sort of question, even if it doesn't fit into the supposed rules for what makes a 'good' question. – Joe Dec 13 '12 at 19:36
9 people who like this question could move their votes to other questions with below 10 votes. Scores over 10 don't help. – Duncan Feb 13 '13 at 0:20
up vote 14 down vote
At what size do I need to start thinking about my data from a "big data" perspective?
added by Chris Travers Oct 6 '12 at 3:14
link
up vote 13 down vote
What data models are best supported by Cassandra? By MongoDB?
added by Chris Gerken Aug 27 '12 at 18:16
link
up vote 11 down vote
With a large-enough data set, will analysis fundamentally differ from random data?
added by warren Sep 5 '12 at 19:04
link
up vote 10 down vote
How can I efficiently anonymize a real-time feed of data into an already anonymized database?
added by Oleksi Jun 15 '12 at 2:18
link
up vote 10 down vote
What are the most efficient algorithms for searching through large data sets?
added by qegal Jul 31 '12 at 4:24
link
This question fits programmers.SE. – reprogrammer Dec 6 '12 at 22:55
up vote 10 down vote
What approach should be taken to partition [census] data across an [HPC cluster]?
added by warren Aug 14 '12 at 16:59
link
up vote 10 down vote
How is google's Dremel different from Hadoop?
added by gpuguy Aug 21 '12 at 14:51
link
This question fits programmers.SE. – reprogrammer Dec 6 '12 at 22:55
up vote 10 down vote
How to preform map-reduce summary from HBase to a RDBMS?
added by Eliran Malka Aug 22 '12 at 22:00
link
Is this a data migration question? – Chris Gerken Aug 27 '12 at 18:17
@ChrisGerken - yes. is there a data migration site on launch? – Eliran Malka Aug 28 '12 at 8:04
up vote 10 down vote
What are some simple algorithms for sampling data with known hot spots for the purpose of driving a hadoop partitioner?
added by Chris Gerken Aug 27 '12 at 18:13
link
up vote 10 down vote
When analyzing "real" data, what effect will random data sets added-in have? Ie, will they skew the results?
added by warren Sep 5 '12 at 19:04
link
up vote 10 down vote
How would you store waveforms and search in the most efficient format to answer questions like Heart Beat > 140 for 5 min.
added by snihalani Sep 8 '12 at 18:50
link
up vote 10 down vote
How to store big graphs eficiently? Example: distance from cities from a big country
added by woliveirajr Nov 21 '12 at 19:02
link
up vote 10 down vote
What methods can I use to efficiently calculate statistics on several terabytes of data?
added by gerrit Dec 11 '12 at 21:01
link
up vote 10 down vote
Is there any special kind of hardware that will make searching through several terabytes of data significantly faster?
added by gerrit Dec 11 '12 at 21:03
link
up vote 10 down vote
How do I efficiently analyse a multi-terabyte dataset located on a far-away computer when I cannot run code where the data are?
added by gerrit Dec 11 '12 at 21:04
link
2  
Why the downvotes? – gerrit Jan 13 '13 at 11:24
2  
Because BigData is not about copying big data files between computers, but about making a huge data set available to users and analysts and connecting it to get more and better information. The downvoters probably thought your problem is too simple and basic for a BigData site. – Chris Jan 22 '13 at 14:46
3  
If colleagues in Japan have a huge dataset and I'm Europe and want to use it, doesn't that fit with making a huge data set available to users? Then the question "how to transfer" is very relevant in practice. – gerrit Jan 22 '13 at 21:52
1  
I've run out of upvotes, but I'd like to see this question on the site. We need to provide good definitive answers to questions that some experts see as simple and basic, as well as the ones they see as complex and hard. – EnergyNumbers Jan 23 '13 at 21:37
2  
Indeed, the dataset I had in mind is exactly the output of a climate model. Not in any nice database, but simply a large number of multi-terabyte files, physically at the other end of the world. To me big data means a lot of data an any format. I hope analysing data stored in terabyte-sized files would be on-topic on Big Data, or I don't really understand the scope... – gerrit Jan 29 '13 at 11:49
show 5 more improvement suggestions
up vote 10 down vote
Is the increase in data outpacing the increase in storage capacity?
added by gerrit Dec 11 '12 at 21:05
link
up vote 10 down vote
Should I care about de-duplication when dealing with bigdata? Any tips on where to set the tradeoff between used space x CPU use on de-duplication ?
added by woliveirajr Dec 13 '12 at 16:13
link
up vote 10 down vote
How will cloud computing affect the development of software for processing big data?
added by Duncan Jan 14 '13 at 12:03
link
up vote 10 down vote
How will audit controls (e.g., HIPAA for healthcare info, SOX, etc) affect storing/processing of big data?
added by Duncan, edited by jonsca Jan 24 '13 at 4:36
link
This question may also work in some form on the Healthcare Stack Exchange – Oleksi Jan 23 '13 at 21:19
up vote 10 down vote
What is common and what is different in various big data domains (eg healthcare, retail, search, telcom, etc)?
added by Duncan Jan 14 '13 at 12:09
link
up vote 10 down vote
Is distributed computing necessary for big data?
added by zkan Jan 17 '13 at 1:56
link
up vote 10 down vote
In the UK, can I use data perturbation to achieve relevant compliance with Data Protection?
added by EnergyNumbers Jan 17 '13 at 17:21
link
2  
This has nothing to do with big data. It is simply a question on UK law. – Ladadadada Feb 3 '13 at 11:07
up vote 10 down vote
What systems can I put in place to ensure that contributors to a big data archive provide suitable metadata?
added by EnergyNumbers Jan 17 '13 at 17:22
link
up vote 10 down vote
How do I avoid getting locked into a proprietary solution when specifying my big data repository?
added by EnergyNumbers Jan 17 '13 at 17:44
link
up vote 10 down vote
How can I store multi-millions of separate image files and be able to reference any one of them at any given time with a reasonable response time?
added by JoshDM Jan 23 '13 at 17:18
link
up vote 10 down vote
How does google corelate can find correlationships in such a short period of time?
added by neo Jan 24 '13 at 17:05
link
up vote 10 down vote
Is there any tool to visualise a time-based geographical data?
added by neo Feb 1 '13 at 8:46
link
1  
timegilder is free and open source tools from mit – jozi Feb 11 '13 at 4:26
up vote 10 down vote
What are good ways of sharing schema-related info between Pig and Hive?
added by Felix GV Feb 4 '13 at 23:54
link
up vote 10 down vote
What platform(s) are "best" for processing Big Data?
added by warren Feb 7 '13 at 16:59
link
3  
I think there's potentially a good question in here, but I'm not sure that this is it yet. How about something along the lines of "how important is feature X when selecting a platform for processing Big Data?" – EnergyNumbers Feb 9 '13 at 8:35
@EnergyNumbers - please feel free to reword to improve :) – warren Feb 9 '13 at 12:36
up vote 10 down vote
Instead of focusing on volume, what are the variety (heterogeneity of data types, representation, and semantic interpretation) aspects of big data?
added by Duncan Feb 13 '13 at 3:32
link
up vote 9 down vote
Is the R programming language a good language to learn if I want to later get into "Big Data" type large-scale data analysis?
added by boulder_ruby May 23 '12 at 16:31
link
up vote 9 down vote
Are there any weaknesses in k-anonymized data that can lead to de-anonymizing data?
added by Oleksi Jun 15 '12 at 2:17
link
up vote 9 down vote
What are the requirements under which which we can use GPUs for real time big data analysis?
added by gpuguy Aug 21 '12 at 14:55
link
up vote 9 down vote
How can I use Hazelcast as a Distributed Cache Mechanism with MangoDB, Hadoop or any other Big data solutions?
added by kamaci Dec 3 '12 at 19:21
link
up vote 9 down vote
What recursion issues in securing big data? Security (eg activity logging) creates tremendous amounts of data that then need to be secured themselves.
added by Duncan Jan 14 '13 at 12:11
link
I think you accidentally this whole sentence. Care to rephrase more clearly? – Felix GV Feb 8 '13 at 17:44
I believe the question is meant to be - "What recursion issues are there in securing big data? – elssar Feb 11 '13 at 6:09
You only get enough characters for the title so I couldn't put all the detail that goes in the body of the question. I say recursion because security of big data creates big data of it's own. Many of the largest data stores where I work are the security data itself. – Duncan Feb 11 '13 at 12:06
up vote 9 down vote
Can big data archives that are paid for by UK research councils be the subject of valid Freedom of Information requests?
added by EnergyNumbers Jan 17 '13 at 17:23
link
2  
This has nothing to do with big data. It is simply a question on UK law. – Ladadadada Feb 3 '13 at 11:01
1  
Big Data actually has a LOT of ethical and juridical implications. Just like GPL can put an end to an otherwise technically sound project, laws and regulations can affect Big Data projects. I don't see a problem with having non-technical questions if the scope of this Q&A site is Big Data in general. There are plenty of GPL-related questions on SO: google.ca/… – Felix GV Feb 8 '13 at 17:42
up vote 9 down vote
How can I harness crowd-sourcing to get the most out of my data repositories, while protecting researchers' and data-owners IP?
added by EnergyNumbers Jan 17 '13 at 17:46
link
1 2 next

This site has been

Closed

This proposal didn't have enough activity during the beta. It was closed 9 years ago.

You could try proposing a variation of this site that you think will have a better chance.

proposed by

warren
106k●95●527●1145

10 years ago

edited by

Robert Cartaino ♦ 3
232k●72●669●1459

10 years ago

viewed

8,654 times

latest activity

9 years ago

followers

users also following

8.5% Freelancing
6.6% Astronomy
6.6% Ebooks
6.6% Open Data
17.2% only this proposal

followers active in

72.8% Stack Overflow
24.5% Super User
23.9% Meta Stack Exchange
20.2% Meta Stack Overflow
19.0% Software Engineering
16.6% Unix & Linux

recent followers

added May 3 '13 at 14:39
Marian
16.4k●2●70●100
added May 3 '13 at 11:30
Saad Bin Akhlaq
51●1
added May 3 '13 at 9:12
mehdi
4,115●3●25●28
added May 2 '13 at 22:00
Bogdan Emil Mariesan
5,777●2●35●63
added May 2 '13 at 18:58
alecxe
493k●143●1242●1612
added May 2 '13 at 16:43
JMK
41.8k●77●305●550
added May 2 '13 at 15:17
Jeroen Landheer
10.1k●3●46●70
added May 1 '13 at 20:44
Judith Moffitt
270k●45●459●857
added May 1 '13 at 18:47
Fh.
54.6k●13●150●335
added Apr 30 '13 at 14:49
MCP_infiltrator
8,175●16●79●150
added Apr 30 '13 at 12:29
Alfe
51●1
added Apr 30 '13 at 10:03
user85869
51●1
added Apr 29 '13 at 21:19
Ryan Gates
9,420●13●105●276
added Apr 29 '13 at 18:38
Chris Ferrie
7,258●1●26●91
added Apr 29 '13 at 7:09
lalit
1,108●2●7●18
added Apr 29 '13 at 5:45
satish
51●1
added Apr 27 '13 at 14:45
AbhishekGirish
2,984●1●22●37
added Apr 26 '13 at 14:17
Remi Thomas
1,679●1●15●27
added Apr 26 '13 at 5:18
y4suyuki
482●3●8
added Apr 25 '13 at 18:39
Daи
29.8k●23●223●504
added Apr 25 '13 at 12:41
user85563
51●1
added Apr 25 '13 at 7:58
musa
7,000●10●33●76
added Apr 25 '13 at 7:00
Fomite
96.1k●30●353●658
added Apr 25 '13 at 4:19
Free Radical
33.6k●15●138●324
added Apr 23 '13 at 23:46
John Lee
1,319●1●16●22
added Apr 22 '13 at 19:39
user85294
51●1
added Apr 22 '13 at 12:01
menorki manil
51●1
added Apr 22 '13 at 11:26
user85254
51●1
added Apr 22 '13 at 9:34
Firdous
6,511●14●54●113
added Apr 22 '13 at 4:41
superuser0
4,419●5●37●90
added Apr 22 '13 at 0:10
khoomeister
4,954●5●33●49
added Apr 20 '13 at 17:28
user85176
51●1
added Apr 19 '13 at 21:29
user85141
51●1
added Apr 19 '13 at 13:59
user85103
51●1
added Apr 19 '13 at 12:31
Tom Morris
12.2k●37●82
added Apr 19 '13 at 12:01
alex
3,116●1●27●46
added Apr 19 '13 at 8:54
mila_frerichs
51●2
added Apr 19 '13 at 7:18
Hellboundz
51●1
added Apr 18 '13 at 15:03
nodnarb
51●2
added Apr 18 '13 at 9:26
AngocA
8,980●7●47●92
feed icon proposal feed
faq | blog | legal | privacy policy | contact us | feedback always welcome
site design / logo © 2023 Stack Exchange Inc; user contributions licensed under cc by-sa; see the licensing help page for more information.
rev 2023.2.15.615