Thursday, April 11, 2013

Grails searchable pluggin to your maven structured project


Grails searchable pluggin gives you capability to add search functionality to your grails project. With this pluggin you can index your domain classes.  It is built on the  Compass Search Engine Framework and Apache Lucene.
The main advantage I experienced while using lucene indexing for searching is, it is simpler and faster than doing  %LIKE%  query on relational database. The underline method of tokenizing words and indexing with lucene can be found from here. Here I explain sample codes I used in my maven structured grails project.
Maven dependencies you need.

I assume There is a Domain class ‘Book’ and a Controller ‘BookController’ and a method ‘search’ in BookController class.

This is a part of the list.gsp where the books are listed.
Domain class
Controller class
Service class
helpful link





Posted by at No comments:
Labels: , , ,

Sunday, February 24, 2013

Apache Shiro Integration for Grails - maven dependency

If your grails project is structured as a maven project, when you are integrating Apache shiro by installing Apache shiro grails plugin as mentioned in http://grails.org/plugin/shiro you need to add the following maven dependency to install Shiro plugin in to you maven structured project.

                                                   <dependency>
                                                           <groupId>org.grails.plugins</groupId>
                                                           <artifactId>shiro</artifactId>
                                                           <version>1.1.4</version>
                                                           <type>zip</type>
                                                           <scope>compile</scope>
                                                 </dependency>
Posted by at No comments:
Labels: ,

Sunday, February 10, 2013

WADL-Web Application Description Language


WADL (Web Application Description Language) is a machine-readable description of HTTP-based REST web Services. 

All WADL elements have the following XML namespace name:http://wadl.dev.java.net/2009/02.
Use the following symbol meanings below.  *  => 0 or more, ? => 0 or 1,  + => 1 or more
<application>
forms the root of a WADL description
<doc>   ( + )
·         Each WADL-defined element can have child doc elements that can be used to document that element.
·         Attributes
xml:lang > Defines the language for the title attribute value and the contents of the doc element
title > A short plain text description of the element being documented
<grammars> (?)
·         Acts as a container for definitions of the format of data exchanged during execution of the protocol described by the WADL document.
·         XML grammars used by the service,
·         Currently specify use of W3C XML Schema or RelaxNG
<include> (*)
allows the definitions of one or more data format descriptions to be included by reference
<resources> (?)
acts as a container for the resources provided by the application
<resource> (+)
·         describes a set of resources
·         Identified by a URI template
·         Specify Web resource and which methods are supported
·         Attributes: id, path, type, queryType
<method> (+)
describes the input to and output from an HTTP protocol method that may be applied to a resource

<request> (?)
the input to the method as a collection of parameters
<response> (?)
describe the possible outputs of the method
<representation> (*)
·         Describe the format of a HTTP entity
·         Can refer to grammars

for a complete description visit w3spec

WADL Document Structure


<application>
<doc/>*
<grammars/>?
<resources base='anyURI'>?
<doc/>*
<resource path='template' type='anyURI+'?>+
<doc/>*
<param/>*
( <method/> | <resource/> )+
</resource>
</resources>
( <method/> | <representation/> | <fault/> |
<resource_type/>)*
</application>

WADL Method Structure 


<method name='NMTOKEN'? id='ID'? href='anyURI'?>
<doc/>*
<request>?
<param>*
<representation/>*
</request>
<response>?
( <representation/> | <fault/> )*
</response>
</method>

Example WADL Description


The following is an example of a WADL description for the Yahoo News Search application (link) 


Try a real REST service 


This is a sample .wadl descriptor of a real REST service. To understand how the path for a web resource of a REST service is can be generated according to the information in .wadl you can try invoking some services (this web service is secured so that you can't access all resources without permission, just try some GET services ). eg: try ttp://repo.jfrog.org/artifactory/api/plugins/getPluginInfo

wadl2java


This tool will generate a java client based on the provided WADL that makes use of the Jersey 1.x and JAX-RS 2.0 client API. Visit open source project 
Most of the IDEs support creating REST services and clients(stub). eg: Guide lines for creating rest services and clients with IntellijIDEA can be found here

Yahoo News Search Stub


public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
public DataSource getAsApplicationXml(
String appid, String query) {...}
public DataSource getAsApplicationJson(
String appid, String query) {...}
public DataSource getAsApplicationPhp(
String appid, String query) {...}
...
}

Maping WADL to Java


public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
}
<resource path="newsSearch">
             <param name="appid" style="query"/>
             <method name="GET">
             ...
            </method>
</resource>
public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
}
<resource path="newsSearch">
<param name="appid" style="query"/>
<method name="GET">
...
</method>
</resource>
public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
}
<resource path="newsSearch">
<param name="appid" style="query"/>
<method name="GET">
...
</method>
</resource>
public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
}
<method name="GET">
<request>
<param name="query" style="query"/>
</request>
<response>
<representation element="y:ResultSet"/>
</response>
</method>
public class NewsSearch {
public NewsSearch() {...}
public ResultSet getAsResultSet(
String appid, String query) {...}
}
<method name="GET">
<request>
<param name="query" style="query"/>
</request>
<response>
<representation element="y:ResultSet"/>
</response>
</method>

Client Code


NewsSearch s = new NewsSearch();
ResultSet rs = s.getAsResultSet("some_app_id","java");
for (Result r: rs.getResultList()) {
System.out.printf("%s (%s)\n",
r.getTitle(),
r.getClickUrl());
}








Posted by at No comments:
Labels: ,

Thursday, September 20, 2012

Counter columns in Cassandra(with CLI,CQL and HECTOR)

This post describes about distributed counters which is a feature of Cassandra versions>0.8.0. Counters in Cassandra allows to keep counting values or keep a sum of some of something. This increment value can be any positive or negative value. Specialty of counters in Cassandra is reading the current value and incrementing it to the new value in a single column value, both together happen as a single atomic operation.

Creating counter columns with CLI and CQL.

Following are the basic commands for counter column
list counters in a row

Counters with CLI


 creating counter columns:
[default@unknown] create keyspace test;
[default@unknown] use test;
[default@test] create column family counters with default_validation_class=CounterColumnType and key_validation_class=UTF8Type and comparator=UTF8Type;

counter operations
increment counter           :     incr counters[row-key][c1];
increment by a value      :    incr counters[row-key][c2] by 3;
decrements counter         :     decr counters[row-key][c2];
decrements by a value    :    decr counters[row-key][c2] by 4;

list counters all in counter column family      :    list counters;
list counters in a row                                     :    get counters[row-key];

delete a counter column:   del counters[row-key][c1];

Counters with CQL


following CQL commands can be used considering above created counter columns.

cqlsh> UPDATE counters SET c1 = c1 + 3, c2 = c2 - 4 WHERE key = row-key;
cqlsh> select * from counters where key=row-key;
    KEY | c1 | c2 |
    row2 |  3 | -4 |

following are more references for using counters with CLI and CQL

link 1    link 2    link 3

Counter columns with hector.


Creating counter columns


Inserting counter column dynamically with Hector


Increment a value in a cell(here mutator is used for batch wise counter value updating) 

Querying in counter column families.


There are limitations counter column families .
  1. Counter column families can not be indexed.
  2. Query operations can be done only for the key in a row but not for counter columns.
for an example lets consider following scenario.

column family= cf1
counter column=Ccolumn1,Ccolum2

KEY
Ccolumn1
Ccolumn2
2341
4
5
KEY
Ccolumn1
Ccolumn2
2345
6
7


queries are limited as below.

      SELECT * FROM cf1 WHERE Ccolumn1>3      - Not allowed
      SELECT * FROM cf1 WHERE KEY > 2341      - Allowed
      SELECT * FROM cf1 WHERE KEY = 2341      - Allowed

Follow the following links for more details.
     link 1     link 2 




Posted by at 3 comments:
Labels: , , , ,

Friday, June 15, 2012

Bulk-loading external data to Cassandra

using sstableloader


sstableloader tool can be used to load data in to Cassandra cluster in batch wise. First we need to generate sstable by reading the external data resource (in this example below from .csv file) and sstableloader stream sstable data files to Cassandra cluster confirming the replication stragegy of the cluster . Following is as example code for loading data from .csv files to Cassandra.


  1. Download and Install Cassandra from here  and use  Cassandra GUI  to monitor your cluster 
  2. Lets consider a csv file with fields  uuid, firstname, lastname, password, age, email
5bd8c586-ae44-11e0-97b8-0026b0ea8cd0, Alice, Smith, asmi1975, 32, alice.smith@mail.com
4bd8cb58-ae44-12e0-a2b8-0026b0ed9cd1, Bob, Miller, af3!df8, 28, bob.miller@mail.com
1ce7cb58-ae44-12e0-a2b8-0026b0ad21ab, Carol, White, cw1845?, 49, c.white@mail.com

Lets consider creating one column family  'Users' with this csv. So create the keyspace 'Demo'  and column family 'Users'.
create keyspace Demo;
use Demo;
create column family Users
    with key_validation_class=LexicalUUIDType
    and comparator=AsciiType
    and column_metadata=[
        { column_name: 'firstname', validation_class: AsciiType }
        { column_name: 'lastname', validation_class: AsciiType }
        { column_name: 'password', validation_class: AsciiType }
        { column_name: 'age', validation_class: LongType, index_type: KEYS }
        { column_name: 'email', validation_class: AsciiType }];

  3.  Create sstable using SSTableSimpleUnsortedWriter class 

configuration - To compile this file the Cassandra jar (>= 0.8.2) needs to be in the classpath (javac -cp <path_to>/apache-cassandra-1.1.1.jar DataImportExample.java). To run it, the Cassandra jar needs to be present as well as the jar of the librairies used by Cassandra (those in the lib/ directory of Cassandra source tree). Validcassandra.yaml and log4j configuration files should also be accessible; typically, this means the conf/directory of the Cassandra source tree should be in the classpath.

 - If you are using Eclipse IDE add the Cassandra/conf folder path to classpath-->Advanced settings-->Add External Folder--> then add apache-cassandra-x.x.x/conf folder in to path.
- In Intellij Idea , got to Run-->Edit Configurations-->Application-->Configuration-->VM Options. There you should give the path to cassandra.yaml as follows.
-Dcassandra-foreground -Dcassandra.config=file:///<path to/apache-cassandra-1.1.2/conf/cassandra.yaml> -ea -Xmx1G


After you run this you will see a folder called Demo is created and there you can find sstables created with .db and .sh1 files.
for an example in the above case it will create sstables as below (when keySpace="Demo" and Column family = "Users")
Demo-Users-hd-1-Data.db
Demo-Users-hd-1-Digest.sha1
Demo-Users-hd-1-Filter.db
Demo-Users-hd-1.index.db
Demo-Users-hd-1.Statistics.db

4. Loading sstables to Cassendra using sstableLoader.

To load sstables from command line   

  •  Go to CASSANDRA_HOME/bin  then , run this command ,  ./sstableloader Demo  or 
  •  Go to CASSANDRA_HOME and run command, bin/sstableloader/some/path/to/the<KeyspaceName>
  • If you are running the all from localhost try the following steps,

     -   Set another loop-back address with command(linix)  :  sudo ifconfig lo:0 127.0.0.2 netmask 255.0.0.0 up
     -   Get a copy of the running Cassandra folder and do the following configuratons.
         Set the rpc address and listen address in  /conf/casandra.yaml to 127.0.0.2. if you want to listen all   interfaces you can set rpc address to 0.0.0.0 
     -   run sstableloader from the copied Cassandra with using command
         ./sstableloader -d 127.0.0.2 <path to sstable../keyspace_name/columnfamily_name>
        Relevant to above example the path should be given as ./sstableloader -d 127.0.0.2 pathtosstables/Demo/Users

 To run sstableloader from java module


 If you want to run sstableLoader through your java module you can run following class. " org.apache.cassandra.tools.BulkLoader.main(args) ".Where "args" will be array of values you supply during the run of sstableloader from command line.

load sstables using  JMX client

with my personal experience using JMX client is much faster than using sstableloader

Posted by at 20 comments:
Labels: , , ,
Subscribe to: Posts (Atom)