This talk will contain 2 topics: 1) How to monitor your MongoDB cluster and what to look out for to prevent explosions. 2) How to build a redundant, scalable queuing system using MongoDB.At Boxed Ice we throw 3.5TB of data into MongoDB each month, which results in processing billions of documents. Fun times
David is an entrepreneurial programmer based in the UK. He is currently working on a server performance monitoring tool, Server Density, through his startup, Boxed Ice.
My Notes:
Boxed Ice is using RabbitMQ for alerts on background processing. But RabbitMQ has no native failover. This was the primary reason they are exploring using MongoDB to provide a persistent data store for recovery.
Basically, they wanted the following:
- Redundancy
- Atomicity
- Speed
- Garbage collection
They chose to use MongoDB over RabbitMQ based on experience working with mongo versus RabbitMQ Not wanting to add yet another system to learn, they decided to use MongoDB exclusively.
Next the presenter talked about monitoring performance in MongoDB.
In memory is always faster than disk. MongoDB has an explain method, use this to determine indexes and disk i/o -- whether operations are reading/writing from/to disk.
Regarding storage, MongoDB pre-allocates in 2gb increments.
When considering sharding, max size assumes capacity same on all nodes. Best to set to 70% of memory capacity.
Rotate your logs, don't let them get too big.
Use journaling and don't go over max 1gb.
To determine used connections: db.serverStatus(). Always use use connection pooling.
connPoolStats
indexCounters, from db.ServerStatus()
Op counters, fsynch setting, config slaves to handle reads
background flushing
Dur
rs.Status() (replicaste set status)
myStatus
Optime, last updated
heartbeat, last comm with members
An overview of the mongostats command presented.
Concerning the value of faults, high values implies not enough ram for indexes/data.
If status is 'locked', this causes queuing and signifies an index miss. Excessive queuing will cause performance issues.
Other useful mongostats commands:
db.currentOp()
db.kill
Boxed Ice runs a site called mongomonitor.com that offers a DB monitoring service for MongoDB instances:
http://www.serverdensity.com/mongodb-monitoring/
In summary:
- Keep indexes in ram and as much data as possible.
- Watch storage usage, both disk and ram.
- Monitor status
Questions from audience/me:
Rabbitmq vs MongoDB thru put?
5000 msg/sec vs 2000 msg/sec
Order of magnitude slower, but not bad.
Someone from the audience asked whether the presenter was aware of something called 'Celery'. Celery offers a way to cluster RabbitMQ instance using MongoDB as a data store.
Looking to store and query location data? MongoDB has you covered. Learn how to structure, and even shard your geo data, along with an unlikely use case: an infinitely large board game!
Eliot is CTO of 10gen, the company that sponsors the open source MongoDB project. Eliot is one of the core MongoDB kernel committers. Eliot is also the co-founder and chief scientist of ShopWiki. In January 2005, he began developing the crawling and data extraction algorithm that is the core of ShopWiki's innovative technology. Eliot has quickly become one of Silicon Alley's up and coming entrepreneurs, having been selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 in 2006. Prior to ShopWiki, Eliot was a software developer in the R&D group at DoubleClick. Eliot received a B.S. in Computer Science from Brown University.
My Notes:
Missed this presentation because I attended the geospatial talk...
The Spring Data project provides sophisticated support for NoSQL datastores. The MongoDB module consists of a namespace to easily setup MongoDB access, a template class to provide a nice API to persist and query objects as well as sophisticated support to build repositories accessing entities stored in a MongoDB. The talk will introduce the Spring Data MongoDB support and present the features in hands on demos.
Eliot is CTO of 10gen, the company that sponsors the open source MongoDB project. Eliot is one of the core MongoDB kernel committers. Eliot is also the co-founder and chief scientist of ShopWiki. In January 2005, he began developing the crawling and data extraction algorithm that is the core of ShopWiki's innovative technology. Eliot has quickly become one of Silicon Alley's up and coming entrepreneurs, having been selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 in 2006. Prior to ShopWiki, Eliot was a software developer in the R&D group at DoubleClick. Eliot received a B.S. in Computer Science from Brown University.
Greg works on various aspects of the MongoDB core server. Prior to 10gen, he completed a PhD at the University of Sussex and Masters at Cornell where he studied multi-agent simulation and computational scaling through assembly. He began his career working on various projects at IBM related to enterprise configuration and modeling, and legacy systems integration.
My Notes:
Rich set of storage and query of geospatial coordinates. Spherical coordinates are the most accurate. Version 1.9 supports multiple locations within a document.
In this talk we will discuss a new design pattern for building applications that consist of many small apps that work together to appear as one website. Cloud Foundry is a PaaS that supports many languages and frameworks as well as many services and data stores, one of which is MongoDB, We will talk about a new pattern enabled by this architecture where you can write different pieces of your application in multiple different languages or frameworks and use a shared MongoDB instance provided by Cloud Foundry's data service layer as an integration point between all the app parts. Not only can you store data in MongoDB for display on the web pages of your app, but you can use MongoDB as a message queue or logging device between apps as well as a shared data structure container. So with this pattern you could have written your main application ui in Ruby on Rails but you may have a few web services or applets written in sinatra or node.js as well as perhaps a Spring java application doing some heavier number crunching or data processing. And use Mongodb via Cloud Foundry's Services layer as an integration point between all the pieces of this application in order to make it appear as one app to the outside world.
Jump on board to learn about combining two of the most exciting technologies to quickly build realtime apps yourself. This talk will introduce the popular Node.js library, Mongoose, which is a MongoDB "ORM" for Node.js. First, the speaker will deliver a quick primer on Node.js. Then, he'll walk you through Mongoose's schema api, powerful query builder, middleware capabilities, and exciting plugin ecosystem. Finally, he'll demonstrate some realtime capabilities using Node.js and Mongoose.
Brian Noguchi is a software engineer in San Francisco. He is the founder of Shortrr, which helps you save time reading the flood of content online. Prior to that, he was the founder and lead engineer of Trendessence, which sold advanced Twitter analytics solutions for the enterprise. He is currently focused on designing and building a realtime framework on top of Node.js that he hopes will do for realtime app development what Rails and Django did for web app development. He is a core contributor to Mongoose, the popular MongoDB package for Node.js, and he is also the author of several other popular Node.js packages. You can find his open source work on github.
My Notes
This was the most crowded session by far. It was mainly about Mongoose, a MongoDB plugin for nodeJS.
http://blog.learnboost.com/blog/mongoose/
There was a brief introduction to nodeJS. As most in attendance were already familar with nodeJS, not a lot of time was spent here. I suggest you look at the offical web sites and blogs for details:
Who uses nodeJS: yammer/github/netflix/learnboost
nodejs = realtime, evented , non-blocking I/O
nodeJS is fast, although it's JavaScript, server side js
One advantage is it can share code between server and browser. Not sure I get that though. Aren't those very different kinds of concerns? Can UI developers use server based algorithms?
nodeJS is an active community
lots of packages supported
recommended packages: express, jade, socketio, mongoose
express=sinatra
jade=template engine http://jade-lang.com/
socketio, name implies it's use http://socket.io/
mongoose= package to support mongoDB which was developed by learnboost http://blog.learnboost.com/blog/mongoose/
mongoose, gives casting, validation, and crud methods
mongoose.schema
crud
c,. save
r, .find, .findall, .findone
u, .find and .save
d, .remove
schema types
string
number, increment, decrement both atomic
objectid
arrays js array syntax, push, pop
comments[CommentSchema] embedded documents!
defaults
validations required:true, enums
custom validations by passing closures/functions like validate Title, 'myTitleValidate')
indexes
index:true or unique:true
nested documents
virtuals
schema.virtual('name.full')
return first+last
advanced querying
mongoose where.[].where[]
namedScopes
scope.name.query
dynamic named scopes defines functions which accept parameters
middleware
pre/post event methods are passed a callback to execute before a save and after a save
need to call next() to maintain atomicity
schema.connectSet for replica sets
.explain is supported
Question from audience: geospatial supported? use chain builder
plugins
mongoose-types (email, for email type validations)
mongoose-auth offers facebook, twittter, git etc auth login capabilities
mongoose-solr is in the works
schema.plugin(aplugin, attribute hash..
Mongoose code is available on github learnboost
See the mongoose blog for more details: http://blog.learnboost.com/blog/mongoose/
How do you unit test? Use espresso package: http://visionmedia.github.com/expresso/
Application Design with MongoDB: 4 Examples
Kyle maintains the MongoDB Ruby Driver and supports the Ruby developer community. Previously, Kyle built e-commerce and social networking applications, and he once worked as teacher of languages and literature. Kyle has presented MongoDB in numerous forums, and he's the author of the forthcoming book MongoDB in Action.
last talk of the day...
Quite honestly, this presenter was over animated and went through his presentation deck at lightening like speed. So I didn't really gleam a lot from this session. Whatever notes are here are of little value I fear.
The first sample problem was how to represent a category hierarchy in a document database. The presented explained the problem and how parent child relationships needed to be defined and modified. Then he presented some methods to for querying for ancestor and descendants and how to keep hierarchies up to date. He show some 'tricks' like using:
$positional operator, updates all instances with the db
No comments:
Post a Comment