android – Fetching big amount of data, what is the best way to go?-ThrowExceptions

Exception or error:

I have severals URLs I need to get data from, this should happen in order, one by one. The amount of data returned by requesting those URLs is relatively big. I need to be able to reschedule particular downloads which failed.

What is the best way to go? Shall I use IntentService, Loaders or something else?

Additional note: I would need not only to download, but also post process the data (create tables in db, fill it with data, etc). So DownloadManger can’t be of help here.

How to solve:

I would use an IntentService.

It has a number of advantages that are suitable for your needs, including being able to download the data without your application running and supporting automatic restart of the service using setIntentRedelivery().

You can set a number of identifiers for the particular job, you need to perform using Intent extras, and you can keep track of the progress using SharedPreferences – that way you can also resume the work if it’s been cancelled previously.

###

The easiest way is probably to use the system DownloadManager http://developer.android.com/reference/android/app/DownloadManager.html

(answering from my phone, so please excuse the lack of formatting)

###

I would suggest a service for this. Having service resolves many problems

  • It would allow reporting of progress asynchronously to the application so you can enable or disable a specific gui in application based on the download status of data

  • It will allow you to continue the download even if the user switches to other application or closes the application.

  • Will allow you to establish independent communication with server to prioritize downloads without user interaction.

###

Try a WakefulIntentService for creating a long-running job that uses wakelocks to keep your task alive and running https://github.com/commonsguy/cwac-wakeful .

Also, if your whole app process is getting killed, you may want to look into persisting the task queue to disk, using something like Tape, from Square

###

I think the way to go is loading urls in an array, then starting an AsyncTask, returning a boolean to onPostExecute indicating if the operation has success or not. then, keeping a global int index, you can run the AsyncTask with the next index if success, or the same index otherwise. Here is a pseudocode

private int index=0;
//this array must be loaded with urls
private ArrayList<String> urlsArray;

new MyDownloaderAsyncTask().execute(urlsArray.get(index));


class MyDownloaderAsyncTask extends AsyncTask<String,String,Boolean>{

@Override
doInBackground(String... input){

//downlaod my data is the function which download data and return a boolean
return downloadMyData();

}

@Override
onPostExecute(Boolean result){

if(result)
    new MyDownloaderAsyncTask().execute(urlsArray.get(++index));
else
    new MyDownloaderAsyncTask().execute(urlsArray.get(index));
}


}

hope this help

###

I have just completed an open source library that can do exactly what you need. Using droidQuery, you can do something like this:

$.ajax(new AjaxOptions().url("http://www.example.com")
                        .type("GET")
                        .dataType("JSON")
                        .context(this)
                        .success(new Function() {
                            @Override
                            public void invoke($ droidQuery, Object... params) {
                                //since dataType is JSON, params[0] is a JSONObject
                                JSONObject obj = (JSONObject) params[0];
                                //TODO handle data
                                //TODO start the next ajax task
                            }
                        })
                        .error(new Function() {
                            @Override
                            public void invoke($ droidQuery, Object... params) {
                                AjaxError error = params[0];
                                //TODO adjust error.options before retry:
                                $.ajax(error.request, error.options);
                            }
                        }));

You can specify other data types, which will return different object types, such as JSONObject, String, Document, etc.

###

Similar to @Murtuza Kabul I’d say use a service, but it’s a little complicated than that. We have a similar situation related to constant internet access and updates, although ours places greater focus on keeping the service running. I’ll try to highlight the main features without drowning you in too much detail (and code is owned by the company 😉 )

  1. android.permission.RECEIVE_BOOT_COMPLETED permission and a BroadcastReceiver listening for android.intent.action.BOOT_COMPLETED to poke the service awake.

  2. Don’t link the service to the Activity, you want it running all the time. eg we call context.startService(new Intent(context.getApplicationContext(), OurService.class))

  3. The service class is just a simple class which registers and calls an OurServiceHandler (as in our case we fire off repeated checks and the Handler manages the ‘ticks’)

  4. We have an OurServiceRunnable which is a singleton which is checked and called by the Handler for each test. It protects against overlapping updates. It delegates to an OurServiceWorker to do the actual lifting.

Sounds heavy handed, but you want to ensure that the service is always running, always ticking (via the Handler) but only running a single check at a time. You’re also going to run into database issue if you use the standard SqlLite DbHelper paradigm, as you can’t open the DB on multiple threads and you definitely want the internet access off the main thread. Our hack was a java.util.concurrent.locks.ReentrantLock protecting access to the DB, but you could probably keep DB access on the UI thread and pass DB operations via the Handler.

Beyond this it’s just a matter of keeping the downloads atomic in terms of “get task, download task, complete task” or enabling it to pick up from a failed state eg downloaded OK, attempt to complete.

###

You should take a look at the volley library :

http://www.javacodegeeks.com/2013/06/android-volley-library-example.html

There is also an interesting video of the author that took place at google io 2013 :

http://www.youtube.com/watch?v=yhv8l9F44qo

Mainly because it eases the process of managing a lot of these fastidious tasks that are connection checking, connection interruption, queue management, retry, resume, etc.

Quoting from the javacodegeeks “Advantages of using Volley :

  1. Volley automatically schedule all network requests. It means that Volley will be taking care of all the network requests your app executes for fetching response or image from web.
  2. Volley provides transparent disk and memory caching.
  3. Volley provides powerful cancellation request API. It means that you can cancel a single request or you can set blocks or scopes of requests to cancel.
  4. Volley provides powerful customization abilities.
  5. Volley provides Debugging and tracing tools”

Update from dennisdrew :
For large file, better use a variant of volley which authorize using another http client implementation. This link gives more details :

The volley article about this modification :

http://ogrelab.ikratko.com/android-volley-examples-samples-and-demos/

The github file detail :

https://github.com/ogrebgr/android_volley_examples/blob/master/src/com/github/volley_examples/toolbox/ExtHttpClientStack.java

###

public class FetchDataFromDBThread implements Runnable {

    /*
     * Defines the code to run for this task.
     */
    @Override
    public void run() {
        // Moves the current Thread into the background
        android.os.Process
                .setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);

        FetchDataFromDB();
    }

}

Leave a Reply

Your email address will not be published. Required fields are marked *