UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
Using Spring with NoSQL databases (SpringOne China 2012)
1. Using Spring with NoSQL databases
Chris Richardson,
Author of POJOs in Action, Founder of the original CloudFoundry.com
@crichardson chris.richardson@springsource.com http://plainoldobjects.com/
2. Presentation goal
NoSQL databases: what, why and
how
How Spring Data simplifies the
development of NoSQL applications
9. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
10. Relational databases are great...
• SQL • Well supported
• High-level • JDBC
• Sorting • Hibernate/JPA
• Aggregation • Spring
• ACID semantics • Well understood
• Developers
• Operators
11. ... but they have limitations
• Object/relational impedance mismatch
• Complicated to map rich domain model to relational schema
• Difficult to handle semi-structured data, e.g. varying attributes
• Schema changes
• Extremely difficult/impossible to scale
• Poor performance for some use cases
12. Solution: Spend Money
OR
http://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG
• Hire more DevOps
• Use application-level sharding
• Build your own middleware
• …
http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
16. Solution: Use NewSQL
• Relational databases with SQL and ACID transactions
AND
• New and improved architecture - designed for modern hardware
• Radically better scalability and performance
• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...
17. Future = multi-paradigm data storage for
enterprise applications
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
18. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
19. Redis
• Advanced key-value store K1 V1
• Written in C
K2 V2
• Very fast, e.g. 100K reqs/sec
• Optional persistence ... ...
• Transactions with optimistic locking
• Master-slave replication
• Sharding using client-side consistent hashing
20. Using Redis (via CLI)
Datatypes: redis 127.0.0.1:6379> set foo 1
OK
•Strings redis 127.0.0.1:6379> get foo
•Hashes "1"
•Maps redis 127.0.0.1:6379> sadd myset a
•Lists (integer) 1
•Sets redis 127.0.0.1:6379> sadd myset b
(integer) 1
•Sorted sets redis 127.0.0.1:6379> smembers myset
1) "a"
2) "b"
redis 127.0.0.1:6379> srem myset a
(integer) 1
redis 127.0.0.1:6379> smembers myset
1) "b"
21. Redis use cases
• Replacement for Memcached • Handling tasks that overload an RDBMS
• Session state • Hit counts - INCR
• Cache of data retrieved from system of • Most recent N items - LPUSH and LTRIM
record (SOR)
• Randomly selecting an item –
• Replica of SOR for queries needing high- SRANDMEMBER
performance
• Queuing – Lists with LPOP, RPUSH, ….
• High score tables – Sorted sets and ZINCRBY
• …
22. MongoDB
• Document-oriented database • Geospatial queries
• JSON-style documents: objects, lists, • Grid FS provides file storage
primitives
• Very fast, asynchronous writes
• Schema-less
• Highly scalable and available
• Transaction = update of a single
document
• Rich query language for dynamic queries
23. Data model = Binary JSON documents
Server
Database: Food To Go
Collection: Restaurants
{
"name" : "TGI Fridays",
"type" : ”American",
"serviceArea" : [
"94619",
"94618"
], Sequence of
"openingHours" : [ bytes on disk
{ è fast i/o
"dayOfWeek" : "Wednesday",
"open" : 1730,
"close" : 2230
}
],
"_id" : ObjectId("4bddc2f49d1505567c6220a0")
}
25. MongoDB query by example
{
serviceArea:"94619",
Find a restaurant that
openingHours: { serves the 94619 zip
$elemMatch : { code and is open at 6pm
"dayOfWeek" : "Monday",
"open": {$lte: 1800}, on a Monday
"close": {$gte: 1800}
}
}
}
DBCursor cursor = collection.find(qbeObject);
while (cursor.hasNext()) {
DBObject o = cursor.next();
…
}
27. MongoDB use cases
• Use cases • Who is using it?
• High volume writes • Shutterfly, Foursquare
• Complex data • Bit.ly Intuit
• Semi-structured data • SourceForge, NY Times
• GILT Groupe, Evite,
• SugarCRM
28. Other NoSQL databases
Type Examples
Extensible columns/Column-oriented Hbase
e
SimpleDB, DynamoDB
r it
avo
Cassandra
r f
you
Graph Neo4j
t
t ou
Key-value
I lef Voldemort, Riak
if
Document So rry CouchDb
http://nosql-database.org/ lists 122+ NoSQL databases
29. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
30. Spring Data is here to help
For
NoSQL databases
http://www.springsource.org/spring-data
31. Spring Data sub-projects
• Relational • QueryDSL
• JPA • Big Data
• JDBC Extensions • Hadoop
• NoSQL • HDFS and M/R
• Redis • Hive
• Mongo • Pig
• HBase • Cascading
• Neo4j • Splunk
• Gemfire • Access
• Lucene • REST
32. What you get
• Template classes that hide the boilerplate code
• Auto-generated (generic) repositories for some NOSQL databases
• Java NoSQL mapping
• Cross Store Persistence
• Support in Roo
34. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
35. Redis challenges
• Connection management: need to get and reliably close connections
• Data mapping: application objects Redis binary/strings
• Multiple client libraries with gratuitously different APIs
36. Spring Data for Redis
• Low-level - RedisConnection(Factory) • Connection management
• Supports Jedis, Jredis, Rjc and Srp • Pluggable Java binary conversion
• Insulates client code from underlying • Support classes:
library
• Collections-backed by RedisTemplate
• High-level - RedisTemplate
• Atomic Counters
• Builds on RedisConnection(Factory)
• Support for Redis pub/sub
38. Using RedisConnectionFactory
public class LowLevelRedisTest {
@Autowired private RedisConnectionFactory redisConnectionFactory;
@Test
public void testLowLevel() { Library independent code J
RedisConnection con = null;
try {
con = redisConnectionFactory.getConnection();
byte[] key = "foo".getBytes(); Ugly byte arrays L
byte[] value = "bar".getBytes();
con.set(key, value);
byte[] retrievedValue = con.get(key);
Assert.assertArrayEquals(value, retrievedValue);
} finally {
if (con != null) { con.close(); }
Need to clean up L
}
}
39. Configuring RedisConnectionFactory
@Configuration
public class RedisConfiguration {
@Value("${databaseHostName}")
protected String databaseHostName;
@Bean
public RedisConnectionFactory jedisConnectionFactory() {
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setHostName(databaseHostName);
factory.setPort(6379);
factory.setUsePool(true);
return factory;
}
}
40. High-level API = RedisTemplate
• Builds on RedisConnection(Factory) • Maps Redis exceptions
DataAccessException
• Analogous to JdbcTemplate
• StringRedisTemplate
• Parameterized type
• Extends RedisTemplate<String,
• K - Key type
String>
• V – Value type
• Keys and values are Strings
• Handles Java Key/Value Redis byte[]
41. Using StringRedisTemplate
public class RedisTemplateTest {
@Autowired private StringRedisTemplate stringRedisTemplate; Returns KV type specific interface
@Test
public void testGetAndSet() {
stringRedisTemplate.opsForValue().set("foo", "bar");
assertEquals("bar", stringRedisTemplate.opsForValue().get("foo"));
}
@Test Converts between Strings and byte[]
public void testHashOps() {
stringRedisTemplate.opsForHash().put("myHash", "myKey", "value");
assertEquals("value",
stringRedisTemplate.opsForHash().get("myHash", "myKey"));
assertEquals(Collections.singleton("myKey"),
stringRedisTemplate.opsForHash().keys("myHash"));
assertEquals(Collections.singletonMap("myKey", "value"),
stringRedisTemplate.opsForHash().entries("myHash"));
}
42. Configuring StringRedisTemplate
@Configuration
public class RedisConfiguration {
@Bean
public RedisConnectionFactory jedisConnectionFactory() {
…
}
@Bean
public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) {
StringRedisTemplate template = new StringRedisTemplate();
template.setConnectionFactory(factory);
return template;
}
}
43. RedisTemplate: Java objects binary data
• DefaultSerializer - defaults to JdkSerializationRedisSerializer
• KeySerializer
• ValueSerializer
• HashKeySerializer
• HashValueSerializer
50. Redis caching support
KVs = <prefix + K, V>
Template needs to (de)serialize K and V Sorted set of all keys for clear()
51. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
52. MongoDB API usage patterns
• Create and store Mongo singleton • Queries
• Externalized server host, port etc. • Construct query object
• Inserts/Updates • mongo.getDatabase(…).getCollection(…)
• Map application POJO DBObject • Iterate through Cursor
• Map DBObject application POJO
• mongo.getDatabase(…).getCollection(…)
• Partial document updates
• Configure asynchronous vs. synchronous writes Higher-level than JDBC but still
repetitive, …
53. Spring Data - MongoDB
• MongoTemplate • Map-Reduce integration
• Query, Criteria, and Update DSLs • GridFS support
• Generic repositories
• Querydsl integration
• Cross-store persistence
• GeoSpatial integration
60. Update example
@Repository
public class RestaurantRepository {
public void addMenuItem(String restaurantId,
MenuItem newMenuItem) {
mongoTemplate.updateFirst(
query(where("_id").is(new ObjectId(restaurantId))),
new Update().push("menuItems", newMenuItem),
Restaurant.class);
}
Atomic, in-place update of document
61. Geospatial example 1
case class FriendRecord(id : String,
name : String,
@Component
location : Point)
class MongoFriendService extends FriendService {
@Autowired
var mongoTemplate: MongoTemplate = _
Collection name
@PostConstruct
def createGeoIndex {
val dbo = new BasicDBObject
dbo.put("location", "2d")
mongoTemplate.getCollection("friendRecord").ensureIndex(dbo)
}
Create geospatial 2d index
62. Geospatial example 2 - finding nearby
@Component
class MongoFriendService extends FriendService {
override def findNearbyFriends(request: NearbyFriendsRequest) = {
val location = new Point(request.longitude, request.latitude)
val distance = new Distance(3, Metrics.MILES)
val query = NearQuery.near(location).maxDistance(distance)
val result = mongoTemplate.geoNear(query, classOf[FriendRecord])
val nearby = result.getContent.map(_.getContent)
FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id)))
}
63. Callbacks – access driver API with exception
translation Exceptions are
translated
@Test
public void testDbCallback() {
Restaurant ajanta = makeAjantaRestaurant();
restaurantRepository.add(ajanta);
assertCollectionExists("restaurants2");
}
private Void assertCollectionExists(final String collectionName) {
return mongoTemplate.execute(new DbCallback<Void>(){
@Override
public Void doInDB(DB db) {
Set<String> collectionNames = db.getCollectionNames();
Assert.assertTrue("Missing from " +
collectionNames,
collectionNames.contains(collectionName));
return null;
}});
}
64. Defining a Mongo Generic Repository
public class Person {
private ObjectId id;
private String firstname;
private String lastname;
… getters and setters
}
interface PersonRepository extends MongoRepository<Person, ObjectId> {
List<Person> findByLastname(String lastName);
}
Person p = new Person("John", "Doe");
personRepository.save(p);
Person p2 = personRepository.findOne(p.getId());
List<Person> johnDoes = personRepository.findByLastname("Doe");
assertEquals(1, johnDoes.size());
65. Mongo Repository configuration
<bean>
<mongo:repositories
base-package="net.chrisrichardson.mongodb.example.mongorepository"
mongo-template-ref="mongoTemplate" />
</beans>
Scans classpath looking for subtypes of MongoRepository in the base package
66. Richer mapping
@Document(collection=”people”) Annotations define mapping: @Document, @Id, @Indexed,
public class Person { @PersistanceConstructor, @CompoundIndex, @DBRef,
@GeoSpatialIndexed, @Value
@Id
private ObjectId id; Map fields instead of properties no getters or setters
private String firstname;
required
@Indexed
private String lastname; Non-default constructor
Index generation
@PersistenceConstructor
public Person(String firstname, String lastname) {
this.firstname = firstname;
this.lastname = lastname;
}
….
}
67. Richer mapping configuration
@Configuration
public class MongoExampleConfig extends AbstractMongoConfiguration {
private @Value("#{mongoDbProperties.databaseName}")
String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}")
String mongoDbHost; Defines MongoTemplate bean
@Override
public Mongo mongo() throws Exception {
return new Mongo(mongoDbHost);
}
@Override Configures classpath scanning
public String getDatabaseName() {
return mongoDbDatabase;
}
@Override
public String getMappingBasePackage() {
return Person.class.getPackage().getName();
}
}
68. Support for the QueryDSL project
Generated from Type-safe
domain model class composable queries
QPerson person = QPerson.person;
Predicate predicate =
person.homeAddress.street1.eq("1 High Street")
.and(person.firstname.eq("John"))
List<Person> people = personRepository.findAll(predicate);
assertEquals(1, people.size());
assertPersonEquals(p, people.get(0));
69. Cross-store/polyglot persistence
Person person = new Person(…);
@Entity entityManager.persist(person);
public class Person {
// In Database Person p2 = entityManager.find(…)
@Id private Long id;
private String firstname;
private String lastname;
// In MongoDB
@RelatedDocument private Address address;
{ "_id" : ObjectId(”….."),
"_entity_id" : NumberLong(1),
"_entity_class" : "net.. Person", "_entity_field_name" : "address",
"zip" : "94611", "street1" : "1 High Street", …}
70. Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
71. Using Mongo and Redis with Cloud Foundry
• Create a Mongo or Redis service
• Bind the service to your application
• Access the service
• Via auto-reconfiguration
• Using <cloud:*/> namespace
80. About <cloud:mongo-db-factory/>
Use when multiple services
are bound
<cloud:mongo-db-factory
id="mongoFactory"
service-name="mongo1"
>
<cloud:mongo-options
connections-per-host="..."
]
max-wait-time="..."
/>
</cloud:mongo-db-factory>
81. NoSQL and Caldecott
• Caldecott let’s you tunnel to a NoSQL service
• Use Redis CLI
• Explore database, adhoc operations
• ...
• Use Mongo CLI etc
• Explore database, adhoc operations
• Mongo dump/restore
• ...
82. Summary
• NoSQL databases sometimes offer a combination of:
• Higher scalability and performance
• Schema less, richer data models
• Spring Data simplifies the development of NoSQL applications
• Cloud Foundry supports Mongo and Redis